Skip to content

Commit

Permalink
Update documents
Browse files Browse the repository at this point in the history
fix: #116
  • Loading branch information
karajan1001 committed Apr 23, 2023
1 parent 6464f4d commit 8001fdb
Show file tree
Hide file tree
Showing 5 changed files with 78 additions and 59 deletions.
35 changes: 0 additions & 35 deletions README.md

This file was deleted.

82 changes: 68 additions & 14 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Ossfs
OSSFS
=====

|PyPI| |Status| |Python Version| |License|
Expand Down Expand Up @@ -30,32 +30,84 @@ Ossfs
:target: https://github.com/psf/black
:alt: Black

**OSSFS** is a Python-based interface for file systems that enables interaction with
OSS (Object Storage Service). Through **OSSFS**, users can utilize fsspec's standard
API to operate on OSS objects

Features
--------
Installation
------------

* TODO
You can install *OSSFS* via pip_ from PyPI_:

.. code:: console
Requirements
------------
$ pip install ossfs
* TODO
Up-to-date package also provided through conda-forge distribution:

.. code:: console
Installation
$ conda install -c conda-forge ossfs
Quick Start
------------

You can install *Ossfs* via pip_ from PyPI_:
Here is a simple example of locating and reading a object in OSS.

.. code:: console
.. code:: python
$ pip install ossfs
import ossfs
fs = ossfs.OSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')
fs.ls('/dvc-test-anonymous/LICENSE')
[{'name': '/dvc-test-anonymous/LICENSE',
'Key': '/dvc-test-anonymous/LICENSE',
'type': 'file',
'size': 11357,
'Size': 11357,
'StorageClass': 'OBJECT',
'LastModified': 1622761222}]
with fs.open('/dvc-test-anonymous/LICENSE') as f:
... print(f.readline())
b' Apache License\n'
For more use case and apis please refer to the documentation of `fsspec <https://filesystem-spec.readthedocs.io/en/latest/index.html>`_

Usage
-----
Async OSSFS
------------

Async **OSSFS** is a variant of ossfs that utilizes the third-party async OSS
backend `aiooss2`_, rather than the official sync one, `oss2`_. Async OSSFS
allows for concurrent calls within bulk operations, such as *cat*, *put*, and
*get* etc even from normal code, and enables the direct use of fsspec in async
code without blocking. The usage of async **OSSFS** is similar to the synchronous
variant; one simply needs to replace **OSSFileSystem** with **AioOSSFileSystem**
need to do is replacing the **OSSFileSystem** with the **AioOSSFileSystem**

.. code:: python
import ossfs
fs = ossfs.AioOSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')
print(fs.cat('/dvc-test-anonymous/LICENSE'))
b' Apache License\n'
...
Although `aiooss2`_ is not officially supported, there are still some
features that are currently lacking. However, in tests involving the
*put*/*get* of 1200 small files, the async version of ossfs ran ten times
faster than the synchronous variant (depending on the pool size of the
concurrency).

+-------------------------------------------+------------------------+
| Task | time cost in (seconds) |
+===========================================+========================+
| put 1200 small files via OSSFileSystem | 35.2688 (13.53) |
+-------------------------------------------+------------------------+
| put 1200 small files via AioOSSFileSystem | 2.6060 (1.0) |
+-------------------------------------------+------------------------+
| get 1200 small files via OSSFileSystem | 32.9096 (12.63) |
+-------------------------------------------+------------------------+
| get 1200 small files via AioOSSFileSystem | 3.3497 (1.29) |
+-------------------------------------------+------------------------+

Contributing
------------
Expand All @@ -80,7 +132,9 @@ please `file an issue`_ along with a detailed description.

.. _Apache 2.0 license: https://opensource.org/licenses/Apache-2.0
.. _PyPI: https://pypi.org/
.. _file an issue: https://github.com/karajan1001/ossfs/issues
.. _file an issue: https://github.com/fsspec/ossfs/issues
.. _aiooss2: https://github.com/karajan1001/aiooss2/
.. _oss2: https://pypi.org/project/oss2/
.. _pip: https://pip.pypa.io/
.. github-only
.. _Contributor Guide: CONTRIBUTING.rst
8 changes: 4 additions & 4 deletions src/ossfs/async_oss.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
BaseOSSFileSystem,
)
from .exceptions import translate_oss_error
from .utils import as_progress_handler, async_pretify_info_result
from .utils import as_progress_handler, async_prettify_info_result

if TYPE_CHECKING:
from aiooss2.models import AioGetObjectResult
Expand Down Expand Up @@ -240,7 +240,7 @@ async def _ls_buckets(self, refresh: bool = False) -> List[Dict[str, Any]]:
results = self.dircache[""]
return results

@async_pretify_info_result
@async_prettify_info_result
async def _ls(self, path: str, detail: bool = True, **kwargs):
"""List files in given bucket, or list of buckets.
Expand Down Expand Up @@ -272,7 +272,7 @@ async def _ls(self, path: str, detail: bool = True, **kwargs):
files = await self._ls_buckets(refresh)
return files

@async_pretify_info_result
@async_prettify_info_result
async def _info(self, path: str, **kwargs):
norm_path = self._strip_protocol(path).lstrip("/")
if norm_path == "":
Expand Down Expand Up @@ -430,7 +430,7 @@ async def _get_file(self, rpath: str, lpath: str, **kwargs):
**kwargs,
)

@async_pretify_info_result
@async_prettify_info_result
async def _find(
self,
path: str,
Expand Down
8 changes: 4 additions & 4 deletions src/ossfs/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

from .base import DEFAULT_BLOCK_SIZE, SIMPLE_TRANSFER_THRESHOLD, BaseOSSFileSystem
from .exceptions import translate_oss_error
from .utils import as_progress_handler, pretify_info_result
from .utils import as_progress_handler, prettify_info_result

if TYPE_CHECKING:
from oss2.models import (
Expand Down Expand Up @@ -184,7 +184,7 @@ def _ls_dir(
except oss2.exceptions.AccessDenied:
return []

@pretify_info_result
@prettify_info_result
def ls(self, path: str, detail: bool = True, **kwargs):
connect_timeout = kwargs.pop("connect_timeout", 60)
norm_path = self._strip_protocol(path).strip("/")
Expand All @@ -201,7 +201,7 @@ def ls(self, path: str, detail: bool = True, **kwargs):

return files

@pretify_info_result
@prettify_info_result
def find(
self,
path: str,
Expand Down Expand Up @@ -550,7 +550,7 @@ def pipe_file(self, path: str, value: str, **kwargs):
bucket=bucket,
)

@pretify_info_result
@prettify_info_result
def info(self, path, **kwargs):
norm_path = self._strip_protocol(path).lstrip("/")
if norm_path == "":
Expand Down
4 changes: 2 additions & 2 deletions src/ossfs/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def _format_unify(path: str, result, detail: bool):
return _copy_and_pretify_list(path, result, detail)


def pretify_info_result(func):
def prettify_info_result(func):
"""Make the return values of `ls` and `info` follows the fsspec's standard
Examples:
--------------------------------
Expand All @@ -71,7 +71,7 @@ def wrapper(ossfs, path: str, *args, **kwargs):
return wrapper


def async_pretify_info_result(func):
def async_prettify_info_result(func):
"""Make the return values of async func `ls` and `info` follows the
fsspec's standard
Examples:
Expand Down

0 comments on commit 8001fdb

Please sign in to comment.