Skip to content

Releases: apache/opendal

v0.50.0

18 Sep 09:05
@Xuanwo Xuanwo
v0.50.0
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
4d060f2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.50.0 Latest
Latest

Release List

Name Version Next
core 0.49.2 0.50.0
integrations/dav-server 0.1.0 0.1.1
integrations/fuse3 0.0.6 0.0.7
integrations/object_store 0.47.0 0.47.1
integrations/parquet 0.1.2 0.2.0
integrations/unftp-sbe 0.0.6 0.0.7
bin/oay 0.41.10 0.41.11
bin/ofs 0.0.11 0.0.12
bin/oli 0.41.10 0.41.11
bindings/c 0.44.12 0.44.13
bindings/cpp 0.45.10 0.45.11
bindings/dotnet 0.1.8 0.1.9
bindings/go 0.1.2 0.1.3
bindings/haskell 0.44.10 0.44.11
bindings/java 0.47.2 0.47.3
bindings/lua 0.1.8 0.1.9
bindings/nodejs 0.47.4 0.47.5
bindings/php 0.1.8 0.1.9
bindings/python 0.45.9 0.45.10
bindings/ruby 0.1.8 0.1.9

Upgrade to OpenDAL Core v0.50

Public API

services-postgresql's connect string now supports only URL format

Previously, it supports both URL format and key-value format. After switching the implementation from tokio-postgres to sqlx, the service now supports only the URL format.

list now returns path itself

Previously, list("a/b") would not return a/b even if it does exist. Since v0.50.0, this behavior has been changed. OpenDAL will now return the path itself if it exists. This change applies to all cases, whether the path is a directory or a file.

Refactoring of the metrics-related layer

In OpenDAL v0.50.0, we did a refactor on all metrics-related layers. They are now sharing the same underlying implemenationts. PrometheusLayer, PrometheusClientLayer and MetricsLayer are now have similar public APIs and exactly the same metrics value.

What's Changed

Added

Changed

  • refactor: use sqlx for sql services by @tisonkun in #5040
  • refactor(core)!: Add observe layer as building block by @Xuanwo in #5064
  • refactor(layers/prometheus): rewrite prometheus layer based on observe mod by @koushiro in #5069
  • refactor(bindings/java): replace num_cpus with std::thread::available_parallelism by @miroim in #5080
  • refactor(layers/prometheus): provide builder APIs by @koushiro in #5072
  • refactor(layers/prometheus-client): provide builder APIs by @koushiro in #5073
  • refactor(layers/metrics): rewrite metrics layer using observe layer by @koushiro in #5098

Fixed

  • fix(core): TimeoutLayer now needs enable tokio time by @Xuanwo in #5057
  • fix(core): Fix failed list related tests by @Xuanwo in #5058
  • fix(services/memory): blocking_scan right range by @meteorgan in #5062
  • fix(core/services/mysql): Fix mysql Capability by @jackyyyyyssss in #5067
  • fix: fix rust 1.76 error due to temporary value being dropped by @aawsome in #5071
  • fix(service/fs): error due to temporary value being dropped by @miroim in #5079
  • fix(core/services/hdfs): Fix the HDFS write failure when atomic_write_dir is set by @meteorgan in #5039
  • fix(services/icloud): adjust error handling code to avoid having to write out result type explicitly by @koushiro in #5091
  • fix(services/monoiofs): handle async cancel during file open by @NKID00 in #5094

Docs

CI

  • ci(bindings/go): add golangci-lint by @yuchanns in #5060
  • ci(bindings/zig): Fix build and test of zig on 0.13 by @Xuanwo in #5102
  • ci: Don't publish with all features by @Xuanwo in #5108
  • ci: Fix upload-artifacts doesn't include hidden files by @Xuanwo in #5112

Chore

  • chore(bindings/go): bump ffi and sys version by @shoothzj in #5055
  • chore: Bump backon to 1.0.0 by @Xuanwo in #5056
  • chore(services/rocksdb): fix misuse rocksdb prefix iterator by @meteorgan in #5059
  • chore(README): add Go binding badge by @yuchanns in #5074
  • chore(deps): bump crate-ci/typos from 1.23.6 to 1.24.3 by @dependabot in #5085
  • chore(layers/prometheus-client): export PrometheusClientLayerBuilder type by @koushiro in #5093
  • chore(layers): check the examples when running tests by @koushiro in #5104
  • chore(integrations/parquet): Bump parquet to 53 by @Xuanwo in #5109
  • chore: Bump OpenDAL to 0.50.0 by @Xuanwo in #5110

New Contributors

Full Changelog: v0.49.2...v0.50.0

Contributors

meteorgan, Xuanwo, and 10 other contributors
Assets 2
Loading
0 Join discussion

v0.49.2

29 Aug 07:40
@Xuanwo Xuanwo
v0.49.2
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
9c20590
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.49.2

Release List

Name Version
core 0.49.2
integrations/dav-server 0.1.0
integrations/fuse3 0.0.6
integrations/object_store 0.47.0
integrations/parquet 0.1.2
integrations/unftp-sbe 0.0.6
bin/oay 0.41.10
bin/ofs 0.0.11
bin/oli 0.41.10
bindings/c 0.44.12
bindings/cpp 0.45.10
bindings/dotnet 0.1.8
bindings/go 0.1.2
bindings/haskell 0.44.10
bindings/java 0.47.2
bindings/lua 0.1.8
bindings/nodejs 0.47.4
bindings/php 0.1.8
bindings/python 0.45.9
bindings/ruby 0.1.8

What's Changed

Added

  • feat(ovfs): support read and write by @zjregee in #5016
  • feat(bin/ofs): introduce integrations/cloudfilter for ofs by @ho-229 in #4935
  • feat(integrations/spring): add AutoConfiguration class for Spring Mvc and Webflux by @shoothzj in #5019
  • feat(services/monoiofs): impl read and write, add behavior test by @NKID00 in #4944
  • feat(core/services-s3): support user defined metadata by @haoqixu in #5030
  • feat: align fn root semantics; fix missing root for some services; rm duplicated normalize ops by @yjhmelody in #5035
  • feat(core): expose configs always by @tisonkun in #5034
  • feat(services/monoiofs): append, create_dir, copy and rename by @NKID00 in #5041

Changed

  • refactor(core): new type to print context and reduce allocations by @evenyag in #5021
  • refactor(layers/prometheus-client): remove useless scheme field from PrometheusAccessor and PrometheusMetricWrapper type by @koushiro in #5026
  • refactor(layers/prometheus-client): avoid multiple clone of labels by @koushiro in #5028
  • refactor(core/services-oss): remove the starts_with by @haoqixu in #5036

Fixed

  • fix(layers/prometheus-client): remove duplicated increment_request_total of write operation by @koushiro in #5023
  • fix(services/monoiofs): drop JoinHandle in worker thread by @NKID00 in #5031

CI

  • ci: Add contents write permission for build-website by @Xuanwo in #5017
  • ci: Fix test for service ghac by @Xuanwo in #5018
  • ci(integrations/spring): add spring boot bean load test by @shoothzj in #5032

Chore

New Contributors

Full Changelog: v0.49.1...v0.49.2

Contributors

Xuanwo, evenyag, and 9 other contributors
Assets 2
Loading
0 Join discussion

v0.49.1

20 Aug 07:19
@Xuanwo Xuanwo
v0.49.1
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
468fca7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.49.1

Release List

Name Next
core 0.49.1
integrations/dav-server 0.0.8
integrations/fuse3 0.0.5
integrations/object_store 0.46.1
integrations/parquet 0.1.1
integrations/unftp-sbe 0.0.5
bin/oay 0.41.9
bin/ofs 0.0.10
bin/oli 0.41.9
bindings/c 0.44.11
bindings/cpp 0.45.9
bindings/dotnet 0.1.7
bindings/go 0.1.1
bindings/haskell 0.44.9
bindings/java 0.47.1
bindings/lua 0.1.7
bindings/nodejs 0.47.3
bindings/php 0.1.7
bindings/python 0.45.8
bindings/ruby 0.1.7

What's Changed

Added

  • feat(ovfs): add lookup and unit tests by @zjregee in #4997
  • feat(gcs): allow setting a token directly by @jdockerty in #4978
  • feat(integrations/cloudfilter): introduce behavior tests by @ho-229 in #4973
  • feat(integrations/spring): add spring project module by @shoothzj in #4988
  • feat(fs): expose the metadata for fs services by @Aitozi in #5005
  • feat(ovfs): add file creation and deletion by @zjregee in #5009

Fixed

  • fix(integrations/spring): correct parent artifactId by @shoothzj in #5007
  • fix(bindings/python): Make sure read until EOF by @Bicheka in #4995

Docs

  • docs: Fix version detect in website by @Xuanwo in #5003
  • docs: add branding, license and trademarks to integrations by @PsiACE in #5006
  • docs(integrations/cloudfilter): improve docs and examples by @ho-229 in #5010

CI

  • ci(bindings/python): Fix aws-lc-rs build on arm platforms by @Xuanwo in #5004

Chore

New Contributors

Full Changelog: v0.49.0...v0.49.1

Contributors

Xuanwo, Aitozi, and 7 other contributors
Assets 2
Loading
0 Join discussion

v0.49.0

13 Aug 09:51
@Xuanwo Xuanwo
v0.49.0
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
42c41ea
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.49.0

Release List

Name Next
core 0.49.0
integrations/dav-server 0.0.7
integrations/fuse3 0.0.4
integrations/object_store 0.46.0
integrations/parquet 0.1.0
integrations/unftp-sbe 0.0.4
bin/oay 0.41.8
bin/ofs 0.0.9
bin/oli 0.41.8
bindings/c 0.44.10
bindings/cpp 0.45.8
bindings/dotnet 0.1.6
bindings/go 0.1.0
bindings/haskell 0.44.8
bindings/java 0.47.0
bindings/lua 0.1.6
bindings/nodejs 0.47.2
bindings/php 0.1.6
bindings/python (*) 0.45.8
bindings/ruby 0.1.6
  • (*): We expected to release Python 0.45.8, but due to #5000, we were unable to. We will release it next time.

OpenDAL Core Upgrade to v0.49

Public API

Configurator now returns associated builder instead

Configurator used to return impl Builder, but now it returns associated builder type directly. This will allow users to use the builder in a more flexible way.

impl Configurator for MemoryConfig {
-    fn into_builder(self) -> impl Builder {
+    type Builder = MemoryBuilder;
+    fn into_builder(self) -> Self::Builder {
        MemoryBuilder { config: self }
    }
}

LoggingLayer now accepts LoggingInterceptor

LoggingLayer now accepts LoggingInterceptor trait instead of configuration. This change will allow users to customize the logging behavior more flexibly.

pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static {
    fn log(
        &self,
        info: &AccessorInfo,
        operation: Operation,
        context: &[(&str, &str)],
        message: &str,
        err: Option<&Error>,
    );
}

Users can now implement the log in the way they want.

OpenDAL Java Binding Upgrade to v0.47

Breaking change

artifactId of the opendal-java has changed from to opendal to align with the convention of entire OpenDAL project.

<dependencies>
<dependency>
  <groupId>org.apache.opendal</groupId>
-  <artifactId>opendal-java</artifactId>
+  <artifactId>opendal</artifactId>
  <version>${opendal.version}</version>
</dependency>
<dependency>
  <groupId>org.apache.opendal</groupId>
-  <artifactId>opendal-java</artifactId>
+  <artifactId>opendal</artifactId>
  <version>${opendal.version}</version>
  <classifier>${os.detected.classifier}</classifier>
</dependency>
</dependencies>

What's Changed

Added

  • feat(o): Add cargo-o layout by @Xuanwo in #4934
  • feat: impl put_multipart in object_store by @Rachelint in #4793
  • feat: introduce opendal AsyncWriter for parquet integrations by @WenyXu in #4958
  • feat(services/http): implement presigned request for backends without authorization by @NickCao in #4970
  • feat(bindings/python): strip the library for minimum file size by @NickCao in #4971
  • feat(gcs): allow unauthenticated requests by @jdockerty in #4965
  • feat: introduce opendal AsyncReader for parquet integrations by @WenyXu in #4972
  • feat(services/s3): add role_session_name in assume roles by @nerdroychan in #4981
  • feat: support root path for moka and mini-moka by @meteorgan in #4984
  • feat(ovfs): export VirtioFs struct by @zjregee in #4983
  • feat(core)!: implement an interceptor for the logging layer by @evenyag in #4961
  • feat(ovfs): support getattr and setattr by @zjregee in #4987

Changed

  • refactor(java)!: Rename artifacts id opendal-java to opendal by @Xuanwo in #4957
  • refactor(core)!: Return associated builder instead by @Xuanwo in #4968
  • refactor(raw): Merge all operations into one enum by @Xuanwo in #4977
  • refactor(core): Use kv based context to avoid allocations by @Xuanwo in #4986

Fixed

Docs

  • docs: nominate-committer add announcement template by @tisonkun in #4954

CI

Chore

New Contributors

Full Changelog: v0.48.0...v0.49.0

Contributors

meteorgan, Xuanwo, and 13 other contributors
Assets 2
Loading
0 Join discussion

v0.48.0

29 Jul 12:35
@Xuanwo Xuanwo
v0.48.0
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
aee7446
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.48.0

Release List

Name Version
core 0.48.0
integrations/cloudfilter 0.0.0
integrations/dav-server 0.0.6
integrations/fuse3 0.0.3
integrations/object_store 0.45.0
integrations/unftp-sbe 0.0.3
bin/oay 0.41.7
bin/ofs 0.0.8
bin/oli 0.41.7
bindings/c 0.44.9
bindings/cpp 0.45.7
bindings/dotnet 0.1.5
bindings/haskell 0.44.7
bindings/java 0.46.4
bindings/lua 0.1.5
bindings/nodejs 0.47.1
bindings/php 0.1.5
bindings/python 0.45.7
bindings/ruby 0.1.5

OpenDAL Core Upgrade to v0.48

Public API

Typo in customized_credential_load

Since v0.48, the customed_credential_load function has been renamed to customized_credential_load to fix the typo of customized.

- builder.customed_credential_load(v);
+ builder.customized_credential_load(v);

S3 service rename security_token to session_token

In 2014 Amazon switched from AWS_SECURITY_TOKEN to AWS_SESSION_TOKEN. To be consistent with the naming of AWS STS, we have renamed the security_token field to session_token in the S3 service.

- builder.security_token(v);
+ builder.session_token(v);

Operator from_iter and via_iter replaces from_map and via_map

Since v0.48, Operator's new APIs from_iter and via_iter methods have deprecated the from_map and via_map methods.

- Operator::from_map::<Fs>(map)?.finish();
+ Operator::from_iter::<Fs>(map)?.finish();

New API from_iter and via_iter should cover all use cases of from_map and via_map.

Service builder now takes ownership

Since v0.48, all service builder now takes ownership self instead of &mut self. This change will allow users to configure the service in a more flexible way.

- let mut builder = S3::default();
- builder.bucket("test");
- builder.root("/path/to/root");
+ let builder = S3::default().bucket("test").root("/path/to/root");
  let op = Operator::new(builder)?.finish();

Raw API

oio::Write::write will write the whole buffer

Starting from version 0.48, oio::Write::write now writes the entire buffer. This update aligns the API more closely with oio::Read::read and simplifies the implementation of concurrent writing.

  trait Write {
-     fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<usize>>;
+     fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<()>>;
  }

write will now return Result<()> instead of Result<usize>. The number of bytes written can be obtained from the buffer's length.

Access::metadata() will return Arc<AccessInfo>

Starting from version 0.48, Access::metadata() will return Arc<AccessInfo> instead of AccessInfo. This change is intended to improve performance and reduce memory usage.

  trait Access {
-     fn metadata(&self) -> AccessInfo;
+     fn metadata(&self) -> Arc<AccessInfo>;
  }

MinitraceLayer renamed to FastraceLayer

The MinitraceLayer has been renamed to FastraceLayer to respond to the transition from minitrace to fastrace.

- use opendal::layers::MinitraceLayer;
+ use opendal::layers::FastraceLayer;

Use Configurator to replace Builder::from_config

Since v0.48, the Builder::from_config and Builder::from_map method has been replaced by the Configurator trait. The Configurator trait provides a more flexible and extensible way to configure OpenDAL.

Service implementers should update their code to use the Configurator trait instead:

impl Configurator for MemoryConfig {
    fn into_builder(self) -> impl Builder {
        MemoryBuilder { config: self }
    }
}

impl Builder for MemoryBuilder {
    const SCHEME: Scheme = Scheme::Memory;
    type Config = MemoryConfig;

    fn build(self) -> Result<impl Access> {
        ...
    }
}

What's Changed

Added

  • feat(services/fs): Support fs config by @meteorgan in #4853
  • feat(services): init monoiofs by @NKID00 in #4855
  • feat(core/types): avoid a copy in Buffer::to_bytes() by cloning contiguous bytes by @LDeakin in #4858
  • feat(core): Add object versioning for OSS by @Lzzzzzt in #4870
  • feat: fs add concurrent write by @hoslo in #4817
  • feat(services/s3): Add object versioning for S3 by @Lzzzzzt in #4873
  • feat(integrations/cloudfilter): read only cloud filter by @ho-229 in #4856
  • feat(bindings/go): Add full native support from C to Go. by @yuchanns in #4886
  • feat(bindings/go): add benchmark. by @yuchanns in #4893
  • feat(core): support user defined metadata for oss by @meteorgan in #4881
  • feat(service/fastrace): rename minitrace to fastrace by @andylokandy in #4906
  • feat(prometheus-client): add metric label about root on using PrometheusClientLayer by @flaneur2020 in #4907
  • feat(services/monoiofs): monoio wrapper by @NKID00 in #4885
  • feat(layers/mime-guess): add a layer that can automatically set Content-Type based on the extension in the path. by @czy-29 in #4912
  • feat(core)!: Make config data object by @tisonkun in #4915
  • feat(core)!: from_map is now fallible by @tisonkun in #4917
  • ci(bindings/go): always test against the latest core by @yuchanns in #4913
  • feat(!): Allow users to build operator from config by @Xuanwo in #4919
  • feat: Add from_iter and via_iter for operator by @Xuanwo in #4921

Changed

  • refactor(services/s3)!: renamed security_token to session_token by @Zyyeric in #4875
  • refactor(core)!: Make oio::Write always write all given buffer by @Xuanwo in #4880
  • refactor(core)!: Return Arc<AccessInfo> for metadata by @Lzzzzzt in #4883
  • refactor(core!): Make service builder takes ownership by @Xuanwo in #4922
  • refactor(integrations/cloudfilter): implement Filter instead of SyncFilter by @ho-229 in #4920

Fixed

  • fix(services/s3): NoSuchBucket is a ConfigInvalid for OpenDAL by @tisonkun in #4895
  • fix: oss will not use the port by @Lzzzzzt in #4899

Docs

CI

  • ci: Add issue template and pr template for opendal by @Xuanwo in #4884
  • ci: Remove CI reviewer since it doesn't work by @Xuanwo in #4891

Chore

New Contributors

Full Changelog: v0.47.3...v0.48.0

Contributors

flaneur2020, jbampton, and 13 other contributors
Assets 2
Loading
0 Join discussion

v0.47.3

07 Jul 01:29
@Xuanwo Xuanwo
v0.47.3
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
f9df0fa
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.47.3

Release List

Name Version Next
core 0.47.2 0.47.3
integrations/dav-server 0.0.4 0.0.5
integrations/fuse3 0.0.1 0.0.2
integrations/object_store 0.44.2 0.44.3
integrations/unftp-sbe 0.0.1 0.0.2
bin/oay 0.41.5 0.41.6
bin/ofs 0.0.6 0.0.7
bin/oli 0.41.5 0.41.6
bindings/c 0.44.7 0.44.8
bindings/cpp 0.45.5 0.45.6
bindings/dotnet 0.1.3 0.1.4
bindings/haskell 0.44.5 0.44.6
bindings/java 0.46.2 0.46.3
bindings/lua 0.1.3 0.1.4
bindings/nodejs 0.47.0 0.47.0
bindings/php 0.1.3 0.1.4
bindings/python 0.45.5 0.45.6
bindings/ruby 0.1.3 0.1.4

What's Changed

Changed

  • refactor: Move ChunkedWrite logic into WriteContext by @Xuanwo in #4826
  • refactor(services/aliyun-drive): directly implement oio::Write. by @yuchanns in #4821

Fixed

  • fix(integration/object_store): Avoid calling API inside debug by @Xuanwo in #4846
  • fix(integration/object_store): Fix metakey requested is incomplete by @Xuanwo in #4844

Docs

  • docs(integration/unftp-sbe): Polish docs for unftp-sbe by @Xuanwo in #4838
  • docs(bin): Polish README for all bin by @Xuanwo in #4839

Chore

Full Changelog: v0.47.2...v0.47.3

Contributors

Xuanwo, TennyZhuang, and 2 other contributors
Assets 2
Loading
0 Join discussion

v0.47.2

03 Jul 05:08
@Xuanwo Xuanwo
v0.47.2
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
a58e802
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.47.2

Relelase List

Name Version Next
core 0.47.1 0.47.2
integrations/cloudfilter 0.0.0 0.0.0
integrations/dav-server 0.0.3 0.0.4
integrations/fuse3 0.0.0 0.0.1
integrations/object_store 0.44.1 0.44.2
integrations/unftp-sbe 0.0.0 0.0.1
integrations/virtiofs 0.0.0 0.0.0
bin/oay 0.41.4 0.41.5
bin/ofs 0.0.5 0.0.6
bin/oli 0.41.4 0.41.5
bindings/c 0.44.6 0.44.7
bindings/cpp 0.45.4 0.45.5
bindings/dotnet 0.1.2 0.1.3
bindings/go 0.0.0 0.0.0
bindings/haskell 0.44.4 0.44.5
bindings/java 0.46.1 0.46.2
bindings/lua 0.1.2 0.1.3
bindings/nodejs 0.46.2 0.47.0
bindings/ocaml 0.0.0 0.0.0
bindings/php 0.1.2 0.1.3
bindings/python 0.45.4 0.45.5
bindings/ruby 0.1.2 0.1.3
bindings/swift 0.0.0 0.0.0
bindings/zig 0.0.0 0.0.0
  • bindings/nodejs 0.47.0 release failed, we will get it fixed in next release.

Breaking changes

Binding Node.js Public API

Now, the append operation has been removed. You can use below code instead.

op.write("path/to/file", Buffer.from("hello world"), { append: true });

What's Changed

Added

Changed

  • refactor(nodejs)!: Remove append api by @bxb100 in #4796
  • refactor(core): Remove unused layer MadsimLayer by @zzzk1 in #4788

Fixed

  • fix(services/aliyun-drive): list dir without trailing slash by @yuchanns in #4766
  • fix(unftp-sbe): remove buffer for get by @George-Miao in #4775
  • fix(services/aliyun-drive): write op cannot overwrite existing files by @yuchanns in #4781
  • fix(core/services/onedrive): remove @odata.count for onedrive list op by @imWildCat in #4803
  • fix(core): Gcs's RangeWrite doesn't support concurrent write by @Xuanwo in #4806
  • fix(tests/behavior): skip test of write_with_overwrite for ghac by @yuchanns in #4823
  • fix(docs): some typos in website and nodejs binding docs by @suyanhanx in #4814
  • fix(core/aliyun_drive): Fix write_multi_max_size might overflow by @Xuanwo in #4830

Docs

CI

Chore

New Contributors

Full Changelog: v0.47.1...v0.47.2

Contributors

imWildCat, Xuanwo, and 10 other contributors
Assets 2
Loading
0 Join discussion

v0.47.1

21 Jun 11:00
@Xuanwo Xuanwo
v0.47.1
This tag was signed with the committer’s verified signature.
Xuanwo Xuanwo
GPG key ID: E95B7C7CB352F6C1
Learn about vigilant mode.
c98e81f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.47.1

What's Changed

Added

  • feat(core): sets default chunk_size and sends buffer > chunk_size directly by @evenyag in #4710
  • feat(services): add optional access_token for AliyunDrive by @yuchanns in #4740
  • feat(unftp-sbe): Add integration for unftp-sbe by @George-Miao in #4753

Changed

  • refactor(ofs): Split fuse3 impl into fuse3_opendal by @Xuanwo in #4721
  • refactor(ovfs): Split ovfs impl into virtiofs_opendal by @zjregee in #4723
  • refactor(*): tiny refactor to the Error type by @waynexia in #4737
  • refactor(aliyun-drive): rewrite writer part by @yuchanns in #4744
  • refactor(object_store): Polish implementation details of object_store by @Xuanwo in #4749
  • refactor(dav-server): Polish dav-server integration details by @Xuanwo in #4751
  • refactor(core): Remove unused size for RangeWrite. by @reswqa in #4755

Fixed

  • fix(s3): parse MultipartUploadResponse to check error in body by @waynexia in #4735
  • fix(services/aliyun-drive): unable to list / by @yuchanns in #4754

Docs

CI

  • ci: verify build website on site content changes by @tisonkun in #4712
  • ci: Fix cert for redis and add docs for key maintenance by @Xuanwo in #4718
  • ci(nodejs): Disable services-all on windows by @Xuanwo in #4762

Chore

New Contributors

Full Changelog: v0.47.0...v0.47.1

Contributors

Xuanwo, evenyag, and 8 other contributors
Assets 2
Loading
0 Join discussion

v0.47.0

10 Jun 15:03
@tisonkun tisonkun
v0.47.0
This tag was signed with the committer’s verified signature.
tisonkun tison
GPG key ID: CD8564CB6A21E526
Learn about vigilant mode.
f2aa333
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.47.0

Core - Upgrade to v0.47

Public API

Reader into_xxx APIs

Since v0.47, Reader's into_xxx APIs requires async and returns Result instead.

- let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048);
+ let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048).await?;

Affected API includes:

  • Reader::into_futures_async_read
  • Reader::into_bytes_stream
  • BlockingReader::into_std_read
  • BlockingReader::into_bytes_iterator

Raw API

Bring Streaming Read Back

As explained in core: Bring Streaming Read Back, we do need read streaming back for better performance and low memory usage.

So our oio::Read changed back to streaming read instead:

trait Read {
-  async fn read(&self, offset: u64, size: usize) -> Result<Buffer>;
+  async fn read(&mut self) -> Result<Buffer>;
}

All services and layers should be updated to meet this change.

Java Binding - Upgrade to v0.46

Breaking change

PR-4641 renames async Operator to AsyncOperator and BlockingOperator to Operator.

New features

PR-4626 implements OperatorInputStream and OperatorOutputStream which implements Java's core IO abstractions InputStream and OutputStream. Users can now read/write bytes streamlined without loading/preparing the bytes fully in memory.

What's Changed

Added

  • feat(core/types): change oio::BlockingReader to Arc by @hoslo in #4577
  • fix: format_object_meta should not require metakeys that don't exist by @rebasedming in #4582
  • feat: add checksums to MultiPartComplete by @JWackerbauer in #4580
  • feat(doc): update object_store_opendal README by @hanxuanliang in #4606
  • feat(services/aliyun-drive): support AliyunDrive by @yuchanns in #4585
  • feat(bindings/python): Update type annotations by @3ok in #4630
  • feat: implement OperatorInputStream and OperatorOutputStream by @tisonkun in #4626
  • feat(bench): add buffer benchmark by @zjregee in #4603
  • feat: Add Executor struct and Execute trait by @Xuanwo in #4648
  • feat: Add executor in OpXxx and Operator by @Xuanwo in #4649
  • feat: Implement and refactor concurrent tasks for multipart write by @Xuanwo in #4653
  • feat(core/types): blocking remove_all for object storage based services by @TennyZhuang in #4665
  • feat(core): Streaming reading while chunk is not set by @Xuanwo in #4658
  • feat(core): Add more context in error context by @Xuanwo in #4673
  • feat: init ovfs by @zjregee in #4652
  • feat: Implement retry for streaming based read by @Xuanwo in #4683
  • feat(core): Implement TimeoutLayer for concurrent tasks by @Xuanwo in #4688
  • feat(core): Add reader size check in complete reader by @Xuanwo in #4690
  • feat(core): Azblob supports azure workload identity by @Xuanwo in #4705

Changed

  • refactor(core): Align naming for AccessorDyn by @morristai in #4574
  • refactor(core): core doesn't expose invalid input error anymore by @Xuanwo in #4632
  • refactor(core): Return unexpected error while content incomplete happen by @Xuanwo in #4633
  • refactor(core): Change Read's behavior to ensure it reads the exact size of data by @Xuanwo in #4634
  • refactor(bin/ofs): Fuse API by @ho-229 in #4637
  • refactor(binding/java)!: rename blocking and async operator by @tisonkun in #4641
  • refactor(core): Use concurrent tasks to refactor block write by @Xuanwo in #4692
  • refactor(core): Migrate RangeWrite to ConcurrentTasks by @Xuanwo in #4696

Fixed

  • fix(devcontainer/post_create.sh): change pnpm@stable to pnpm@latest by @GG2002 in #4584
  • fix(bin/ofs): privileged mount crashes when external umount by @ho-229 in #4586
  • fix(bin/ofs): ofs read only mount by @ho-229 in #4602
  • fix(raw): Allow retrying request while decoding response failed by @Xuanwo in #4612
  • fix(core): return None if metadata unavailable by @NKID00 in #4613
  • fix(bindings/python): Use abi3 and increase MSPV to 3.11 by @Xuanwo in #4623
  • fix: Fetch the content length while end_bound is unknown by @Xuanwo in #4631
  • fix: ofs write behavior by @ho-229 in #4617
  • fix(core/types): remove_all not work under object-store backend by @TennyZhuang in #4659
  • fix(ofs): Close file during flush by @Xuanwo in #4680
  • fix(core): RetryLayer could panic when other threads raises panic by @Xuanwo in #4685
  • fix(core/prometheus): Fix metrics from prometheus not correct for reader by @Xuanwo in #4691
  • fix(core/oio): Make ConcurrentTasks cancel safe by only pop after ready by @Xuanwo in #4707

Docs

CI

  • ci: upgrade typos to 1.21.0 and ignore changelog by @hezhizhen in #4601
  • ci: Disable jfrog webdav tests for it's keeping failed by @Xuanwo in #4607
  • ci: use official typos github action by @shoothzj in #4635
  • build(deps): upgrade crc32c to 0.6.6 for nightly toolchain by @tisonkun in #4650

Chore

New Contributors

Full Changelog: v0.46.0...v0.47.0

Contributors

jokester, Xuanwo, and 21 other contributors
Assets 2
Loading

v0.46.0

06 May 01:26
@tisonkun tisonkun
v0.46.0
This tag was signed with the committer’s verified signature.
tisonkun tison
GPG key ID: CD8564CB6A21E526
Learn about vigilant mode.
84586e5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.46.0

Upgrade to v0.46

Public API

MSRV Changed to 1.75

Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT and AFIT.

Services Feature Flag

Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.

Additionally, we have removed all reqwest-related feature flags:

  • Users must now directly use reqwest's feature flags for options like rustls, native-tls, etc.
  • The rustls feature is no longer enabled by default; it must be activated manually.
  • OpenDAL no longer offers the trust-dns option; users should configure the client builder directly.

Range Based Read

Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.

opendal::Reader now have APIs like:

let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;

Buffer Based IO

Since version 0.46, OpenDAL features a native Buffer struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.

OpenDAL's Reader will return Buffer and Writer will accept Buffer as input. Users who have implemented their own IO traits should update their code to use the new Buffer struct.

let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;

let w = op.writer("test.txt").await?;

// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;

// Make sure file has been written completely.
w.close().await?;

To enhance usability, we've integrated bridges into bytes::Buf and bytes::BufMut, allowing users to directly interact with the bytes API.

let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;

let w = op.writer("test.txt").await?;

// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;

// Make sure file has been written completely.
w.close().await?;

Bridge API

OpenDAL's Reader and Writer previously implemented APIs such as AsyncRead and AsyncWrite directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.

Since v0.46, OpenDAL provides bridge APIs for Reader and Writer instead.

let r = op.reader("test.txt").await?;

// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);

let w = op.writer("test.txt").await?;

// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();

Raw API

Async in IO trait

Since version 0.46, OpenDAL has adopted Rust's native async_in_trait for our core IO traits, including oio::Read, oio::Write, and oio::List.

This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.

Users who have implemented their own IO traits should update their code to use the new async trait syntax.

What's Changed

Added

  • feat(services/github): add github contents support by @hoslo in #4281
  • feat: Allow selecting webpki for reqwest by @arlyon in #4303
  • feat(services/swift): add support for storage_url configuration in swift service by @zjregee in #4302
  • feat(services/swift): add ceph test setup for swift by @zjregee in #4307
  • docs(website): add local content search based on lunr plugin by @m1911star in #4348
  • feat(services/sled): add SledConfig by @yufan022 in #4351
  • feat : Implementing config for part of services by @AnuRage-git in #4344
  • feat(bindings/java): explicit async runtime by @tisonkun in #4376
  • feat(services/surrealdb): support surrealdb service by @yufan022 in #4375
  • feat(bindings/java): avoid double dispose NativeObject by @tisonkun in #4377
  • feat : Implement config for services/foundationdb by @AnuRage-git in #4355
  • feat: add ofs ctrl-c exit unmount hook by @oowl in #4393
  • feat: Implement RFC-4382 Range Based Read by @Xuanwo in #4381
  • feat(ofs): rename2 lseek copy_file_range getattr API support by @oowl in #4395
  • feat(services/github): make access_token optional by @hoslo in #4404
  • feat(core/oio): Add readable buf by @Xuanwo in #4407
  • feat(ofs): add freebsd OS support by @oowl in #4403
  • feat(core/raw/oio): Add Writable Buf by @Xuanwo in #4410
  • feat(bin/ofs): Add behavior test for ofs by @ho-229 in #4373
  • feat(core/raw/buf): Reduce one allocation by Arc::from_iter by @Xuanwo in #4440
  • feat: ?Send async trait for HttpBackend when the target is wasm32 by @waynexia in #4444
  • feat: add HttpClient::with() constructor by @waynexia in #4447
  • feat: Move Buffer as public API by @Xuanwo in #4450
  • feat: Optimize buffer implementation and add stream support by @Xuanwo in #4458
  • feat(core): Implement FromIterator for Buffer by @Xuanwo in #4459
  • feat(services/ftp): Support multiple write by @xxxuuu in #4425
  • feat(raw/oio): block write change to buffer by @hoslo in #4466
  • feat(core): Implement read and read_into for Reader by @Xuanwo in #4467
  • feat(core): Implement into_stream for Reader by @Xuanwo in #4473
  • feat(core): Tune buffer operations based on benchmark results by @Xuanwo in #4468
  • feat(raw/oio): Use Buffer as cache in RangeWrite by @reswqa in #4476
  • feat(raw/oio): Use Buffer as cache in OneshotWrite by @reswqa in #4477
  • feat: add list/copy/rename for dropbox by @zjregee in #4424
  • feat(core): Implement write and write_from for Writer by @zjregee in #4482
  • feat(core): Add auto ranged read and concurrent read support by @Xuanwo in #4486
  • feat(core): Implement fetch for Reader by @Xuanwo in #4488
  • feat(core): Allow concurrent reading on bytes stream by @Xuanwo in #4499
  • feat: provide send-wrapper to contidionally implement Send for operators by @waynexia in #4443
  • feat(bin/ofs): privileged mount by @ho-229 in #4507
  • feat(services/compfs): init compfs by @George-Miao in #4519
  • feat(raw/oio): Add PooledBuf to allow reuse buffer by @Xuanwo in #4522
  • feat(services/fs): Add PooledBuf in fs to allow reusing memory by @Xuanwo in #4525
  • feat(core): add access methods for Buffer by @George-Miao in #4530
  • feat(core): implement IoBuf for Buffer by @George-Miao in #4532
  • feat(services/compfs): compio runtime and compfs structure by @George-Miao in #4534
  • feat(core): change Result to default error by @George-Miao in #4535
  • feat(services/github): support list recursive by @hoslo in #4423
  • feat: Add crc32c checksums to S3 Service by @JWackerbauer in #4533
  • feat: Add into_bytes_sink for Writer by @Xuanwo in #4541

Changed

  • refactor(core/raw): Migrate oio::Read to async in trait by @Xuanwo in #4336
  • refactor(core/raw): Align oio::BlockingRead API with oio::Read by @Xuanwo in #4349
  • refactor(core/oio): Migrate oio::List to async fn in trait by @Xuanwo in #4352
  • refactor(core/raw): Migrate oio::Write from WriteBuf to Bytes by @Xuanwo in https://github...
Read more

Contributors

jbampton, mobiusklein, and 26 other contributors
Assets 2
Loading