Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: pickwicksoft/pystreamapi
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.0.1
Choose a base ref
...
head repository: pickwicksoft/pystreamapi
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Aug 14, 2023

  1. Bump joblib from 1.3.1 to 1.3.2

    Bumps [joblib](https://github.com/joblib/joblib) from 1.3.1 to 1.3.2.
    - [Release notes](https://github.com/joblib/joblib/releases)
    - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst)
    - [Commits](joblib/joblib@1.3.1...1.3.2)
    
    ---
    updated-dependencies:
    - dependency-name: joblib
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Aug 14, 2023
    Copy the full SHA
    45fcee5 View commit details

Commits on Aug 16, 2023

  1. Merge pull request #67 from PickwickSoft/dependabot/pip/joblib-1.3.2

    Bump joblib from 1.3.1 to 1.3.2
    garlontas authored Aug 16, 2023
    Copy the full SHA
    682f5b1 View commit details

Commits on Aug 29, 2023

  1. 🧑‍💻 Fix autocomplete not working missing typehint

    Add type hint in wrapper functions for BaseStream. Fixes not working autocomplete.
    garlontas committed Aug 29, 2023
    Copy the full SHA
    bea7f31 View commit details
  2. Merge pull request #68 from PickwickSoft/bugfix/fix-autocomplete-type…

    …hints-decorator
    
    🧑‍💻 Fix autocomplete not working missing typehint
    garlontas authored Aug 29, 2023
    Copy the full SHA
    8340e5f View commit details

Commits on Aug 30, 2023

  1. 🔖 Set version to 1.0.2

    garlontas committed Aug 30, 2023
    Copy the full SHA
    4ede887 View commit details
  2. Merge pull request #69 from PickwickSoft/release/v1.0.2

    🔖 Set version to 1.0.2
    garlontas authored Aug 30, 2023
    Copy the full SHA
    b305992 View commit details

Commits on Sep 18, 2023

  1. 📝 Create CONTRIBUTING.md

    garlontas committed Sep 18, 2023
    Copy the full SHA
    041666b View commit details
  2. Copy the full SHA
    acecca9 View commit details
  3. Merge pull request #74 from PickwickSoft/wip/add-contributing-guidelines

    📝 Create CONTRIBUTING.md
    garlontas authored Sep 18, 2023
    Copy the full SHA
    0dd9218 View commit details

Commits on Sep 19, 2023

  1. Copy the full SHA
    2161175 View commit details
  2. Merge pull request #75 from PickwickSoft/documentation/contributing-f…

    …ix-pylint-link
    
    Update pylint link in CONTRIBUTING.md
    garlontas authored Sep 19, 2023
    Copy the full SHA
    dd00c9e View commit details

Commits on Sep 26, 2023

  1. ⚡ Fix CSV loader very slow

    Improve speed of CSV loader by adding parameter to disable type casting
    garlontas committed Sep 26, 2023
    Copy the full SHA
    4736671 View commit details
  2. Merge pull request #76 from PickwickSoft/bugfix/#63/fix-csv-loader-ve…

    …ry-slow
    
    ⚡ Fix CSV loader very slow
    garlontas authored Sep 26, 2023
    Copy the full SHA
    775b687 View commit details

Commits on Sep 28, 2023

  1. Copy the full SHA
    657db28 View commit details
  2. ✨ Implement JSON loader

    garlontas committed Sep 28, 2023
    Copy the full SHA
    7eb2639 View commit details
  3. 📝 Add JSON loader

    garlontas committed Sep 28, 2023
    Copy the full SHA
    5af2ac6 View commit details
  4. Copy the full SHA
    2b4c3c8 View commit details
  5. refactor: autofix issues in 2 files

    Resolved issues in the following files with DeepSource Autofix:
    1. tests/test_csv_loader.py
    2. tests/test_json_loader.py
    deepsource-autofix[bot] authored Sep 28, 2023
    Copy the full SHA
    828e9e0 View commit details
  6. Copy the full SHA
    561ac17 View commit details
  7. Merge pull request #77 from PickwickSoft/feature/#70/data-loader-for-…

    …json
    
    Feature/#70/data loader for json
    garlontas authored Sep 28, 2023
    Copy the full SHA
    31c8dea View commit details
  8. 🔖 Release Version 1.1.0

    garlontas committed Sep 28, 2023
    Copy the full SHA
    79fc829 View commit details
  9. Merge pull request #78 from PickwickSoft/release/v1.1.0

    🔖 Release Version 1.1.0
    garlontas authored Sep 28, 2023
    Copy the full SHA
    21dcdf1 View commit details

Commits on Dec 30, 2023

  1. Copy the full SHA
    0915548 View commit details
  2. Merge pull request #79 from PickwickSoft/bugfix/date-conditions-edge-…

    …case
    
    🐛 Fix bug in date conditions occuring at the end of a year
    garlontas authored Dec 30, 2023
    Copy the full SHA
    b4fef93 View commit details
  3. Copy the full SHA
    f9a08bf View commit details
  4. Copy the full SHA
    7402296 View commit details
  5. Merge pull request #80 from PickwickSoft/bugfix/fix-code-issues

    Fix code smells and other issues
    garlontas authored Dec 30, 2023
    Copy the full SHA
    9780e7d View commit details
  6. ♻️ Refactor the tests

    garlontas committed Dec 30, 2023
    Copy the full SHA
    151434d View commit details
  7. ✅ Update test

    garlontas committed Dec 30, 2023
    Copy the full SHA
    da77ecb View commit details
  8. ✅ Fix code smell in test

    garlontas committed Dec 30, 2023
    Copy the full SHA
    835d9f0 View commit details
  9. Merge pull request #81 from PickwickSoft/bugfix/fix-code-issues

    ♻️ Refactor the tests
    garlontas authored Dec 30, 2023
    Copy the full SHA
    198d9c2 View commit details
  10. Copy the full SHA
    8ffaaa9 View commit details
  11. 🩹 Fix issues

    garlontas committed Dec 30, 2023
    Copy the full SHA
    36b3fd0 View commit details
  12. Copy the full SHA
    782b541 View commit details
  13. Copy the full SHA
    46ff34b View commit details
  14. ♻️ Change import

    garlontas committed Dec 30, 2023
    Copy the full SHA
    0c3d304 View commit details
  15. 🎨 Format class docstring

    garlontas committed Dec 30, 2023
    Copy the full SHA
    443bbe1 View commit details
  16. Copy the full SHA
    f07586b View commit details
  17. Copy the full SHA
    278b015 View commit details
  18. Copy the full SHA
    42c520c View commit details
  19. Copy the full SHA
    973038c View commit details
  20. Copy the full SHA
    9e76342 View commit details
  21. Merge pull request #82 from PickwickSoft/feature/#72/data-loader-for-xml

    ✨ Create data loader for XML
    garlontas authored Dec 30, 2023
    Copy the full SHA
    92bda09 View commit details
  22. 🚚 Restructure tests

    garlontas committed Dec 30, 2023
    Copy the full SHA
    f821a48 View commit details
  23. Merge pull request #87 from PickwickSoft/wip/refactor-test-structure

    🚚 Restructure tests
    garlontas authored Dec 30, 2023
    Copy the full SHA
    ddc3566 View commit details
  24. Copy the full SHA
    252c96b View commit details
  25. 🎨 Remove unused imports

    garlontas committed Dec 30, 2023
    Copy the full SHA
    cad76c5 View commit details
  26. Merge pull request #88 from PickwickSoft/bugfix/fix-date-edge-case

    🐛 Fix edge case scenarios in date conditions
    garlontas authored Dec 30, 2023
    Copy the full SHA
    30dc953 View commit details

Commits on Feb 22, 2024

  1. add test case

    NuclearMissile committed Feb 22, 2024
    Copy the full SHA
    f224a9c View commit details
  2. fix .parallel() bug

    NuclearMissile committed Feb 22, 2024
    Copy the full SHA
    1449b53 View commit details
Showing with 1,404 additions and 580 deletions.
  1. +2 −2 .github/workflows/unittests.yml
  2. +197 −0 CONTRIBUTING.md
  3. +53 −7 README.md
  4. +222 −314 poetry.lock
  5. +12 −3 pyproject.toml
  6. +1 −1 pystreamapi/__init__.py
  7. +1 −1 pystreamapi/__stream.py
  8. +4 −0 pystreamapi/__stream_converter.py
  9. +40 −4 pystreamapi/_itertools/tools.py
  10. +4 −3 pystreamapi/_parallel/fork_and_join.py
  11. +2 −2 pystreamapi/_parallel/parallelizer.py
  12. +26 −13 pystreamapi/_streams/__base_stream.py
  13. +13 −9 pystreamapi/_streams/__parallel_stream.py
  14. +10 −11 pystreamapi/_streams/__sequential_stream.py
  15. +3 −4 pystreamapi/_streams/error/__error.py
  16. +27 −25 pystreamapi/conditions/date.py
  17. +50 −0 pystreamapi/loaders/__csv/__csv_loader.py
  18. 0 tests/assets/empty.csv → pystreamapi/loaders/__csv/__init__.py
  19. +0 −52 pystreamapi/loaders/__csv_loader.py
  20. +17 −4 pystreamapi/loaders/__init__.py
  21. 0 pystreamapi/loaders/__json/__init__.py
  22. +46 −0 pystreamapi/loaders/__json/__json_loader.py
  23. +24 −0 pystreamapi/loaders/__loader_utils.py
  24. 0 pystreamapi/loaders/__xml/__init__.py
  25. +117 −0 pystreamapi/loaders/__xml/__xml_loader.py
  26. 0 pystreamapi/loaders/__yaml/__init__.py
  27. +56 −0 pystreamapi/loaders/__yaml/__yaml_loader.py
  28. +0 −30 setup.cfg
  29. 0 tests/_conditions/__init__.py
  30. +1 −0 tests/{ → _conditions}/date_test.py
  31. 0 tests/{ → _conditions}/test_combiners_conditions.py
  32. +1 −2 tests/{ → _conditions}/test_date_conditions.py
  33. +1 −1 tests/{ → _conditions}/test_date_conditions_type_date.py
  34. 0 tests/{ → _conditions}/test_numeric_conditions.py
  35. 0 tests/{ → _conditions}/test_string_conditions.py
  36. 0 tests/{ → _conditions}/test_type_conditions.py
  37. 0 tests/_itertools/__init__.py
  38. 0 tests/{ → _itertools}/test_itertools.py
  39. 0 tests/_lazy/__init__.py
  40. 0 tests/{ → _lazy}/helper.py
  41. +1 −1 tests/{ → _lazy}/test_process.py
  42. +1 −1 tests/{ → _lazy}/test_queue.py
  43. 0 tests/_loaders/__init__.py
  44. +3 −0 tests/_loaders/file_test.py
  45. +73 −0 tests/_loaders/test_csv_loader.py
  46. +74 −0 tests/_loaders/test_json_loader.py
  47. +107 −0 tests/_loaders/test_xml_loader.py
  48. +64 −0 tests/_loaders/test_yaml_loader.py
  49. 0 tests/_parallel/__init__.py
  50. 0 tests/{ → _parallel}/test_fork_and_join.py
  51. 0 tests/_streams/__init__.py
  52. 0 tests/_streams/error/__init__.py
  53. +15 −15 tests/{ → _streams/error}/test_error_handler.py
  54. +5 −5 tests/{ → _streams/error}/test_error_handler_streams.py
  55. 0 tests/_streams/numeric/__init__.py
  56. +6 −6 tests/{ → _streams/numeric}/test_numeric_base_stream.py
  57. +1 −1 tests/{ → _streams/numeric}/test_numeric_stream_implementation.py
  58. +81 −7 tests/{ → _streams}/test_base_stream.py
  59. 0 tests/{ → _streams}/test_stream_closed.py
  60. +9 −0 tests/{ → _streams}/test_stream_converter.py
  61. +31 −7 tests/{ → _streams}/test_stream_implementation.py
  62. +0 −3 tests/assets/data.csv
  63. +0 −2 tests/assets/data2.csv
  64. +0 −43 tests/test_loaders.py
  65. +3 −1 tox.ini
4 changes: 2 additions & 2 deletions .github/workflows/unittests.yml
Original file line number Diff line number Diff line change
@@ -64,14 +64,14 @@ jobs:
# Install dependencies. `--no-root` means "install all dependencies but not the project
# itself", which is what you want to avoid caching _your_ code. The `if` statement
# ensures this only runs on a cache miss.
- run: poetry install --no-root
- run: poetry install --no-root --extras "all"
if: steps.cache-deps.outputs.cache-hit != 'true'

# Now install _your_ project. This isn't necessary for many types of projects -- particularly
# things like Django apps don't need this. But it's a good idea since it fully-exercises the
# pyproject.toml and makes that if you add things like console-scripts at some point that
# they'll be installed and working.
- run: poetry install
- run: poetry install --extras "all"

# Runs a single command using the runners shell
- name: Run Unittests
197 changes: 197 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# Contributing to pystreamapi

We are thrilled to have you here! You, the open source contributors, are what makes this project so great. We appreciate
all of your input and contributions to help make pystreamapi the best it can be.

## Ways to contribute

There are many ways to contribute to pystreamapi. Here is how you can help:

- [Report bugs and make feature requests by opening issues](#reporting-bugs-and-feature-requests)
- [Write code and fix/close existing issues](#contributing-code)
- [Improve documentation](#contributing-documentation)

## Important Resources

- [Issue Tracker](https://github.com/PickwickSoft/pystreamapi/issues): Report bugs and make feature requests
- [Documentation](https://pystreamapi.pickwicksoft.org/): Read the documentation
- [Project Board](https://github.com/orgs/PickwickSoft/projects/11): See the current development status
- [GitHub Discussions](https://github.com/PickwickSoft/pystreamapi/discussions): Ask questions and discuss ideas

## Reporting Bugs and Feature Requests

We use GitHub issues to track bugs and feature requests. Please ensure your bug description is clear and has sufficient
instructions to be able to reproduce the issue. If you are requesting a new feature, please explain why you think it is
needed and describe how it should work.

We already created prefilled templates for you to use when creating issues in order to improve the quality of the
information you provide.

Please do not use the issue tracker for personal support requests. Instead,
use [GitHub Discussions](https://github.com/PickwickSoft/pystreamapi/discussions/categories/q-a).

## Branches

The `main` branch is the stable branch. All development work should be done in a separate branch. When you are ready to
submit a pull request, please submit it against the `main` branch.

The `docs` branch is the branch used to build the documentation. It automatically updates the GitBook documentation when
a pull request is merged into it.

## Contributing Code

If you are interested in contributing code to pystreamapi, please follow these steps:

1. [Fork the repository and clone it](#fork-the-repository)
2. [Create a new branch for each feature or improvement](#create-a-new-branch)
3. [Install the development dependencies](#install-development-dependencies)
4. [Make your changes](#make-your-changes)
5. [Test and lint your code](#test-and-lint-your-code)
6. [Commit your changes](#commit-your-changes)
7. [Submit a pull request against the `main` branch]()
8. Wait for your pull request to be reviewed and merged

:tada: Congratulations! You have successfully contributed to pystreamapi!

### Fork the repository

You can fork the repository by clicking on the "Fork" button in the top right corner of the repository page or
by [clicking here](https://github.com/PickwickSoft/pystreamapi/fork). This will create a copy of the repository in your
own GitHub account.

If you need help with forking a repository, please refer to
the [GitHub documentation](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo).

After you have forked the repository, you can [clone](https://help.github.com/articles/cloning-a-repository/) it to your
local machine.

### Create a new branch

Create a new branch for each feature or improvement you are working on. Please follow
our [branch naming conventions](https://github.com/PickwickSoft/conventions/blob/main/BRANCH_NAMING.md).

Create the branch from the `main` branch by running the following command:

```bash
git checkout -b BRANCH_NAME main
```

### Install development dependencies

Install Poetry if you haven't already by following the
instructions [here](https://python-poetry.org/docs/#installation).

Install the development dependencies by running the following command:

```bash
poetry install
```

Set poetry as the default interpreter for your project in your IDE. This will ensure that the correct dependencies are
used when running the project.

Alternatively, you can use the following command to activate the virtual environment:

```bash
poetry shell
```

### Make your changes

Make your changes to the code. Please follow the best practices and conventions for python development. You can find
them on the official style guide for python code: [PEP 8](https://www.python.org/dev/peps/pep-0008/).

### Test and lint your code

#### Testing

Before submitting a pull request, please make sure to write tests and lint the code.

All tests are located in the `tests` directory. Our tests can be executed
using [Coverage.py](https://coverage.readthedocs.io/).

To run the tests, execute the following command in the root directory of the project:

```bash
coverage run --source "pystreamapi/" -m unittest discover -s tests -t tests --pattern 'test_*.py'
```

To generate a coverage report, execute the following command and afterwards click on the link to open the report in your
browser:

```bash
coverage html && cd htmlcov/ && python3 -m http.server
```

Please make sure that all tests pass and the coverage of your code is 100% before submitting a pull request.

#### Linting

We use [pylint](https://pylint.readthedocs.io/en/latest/) to lint our code. You can run pylint by executing the
following command in the
root directory of the project after staging your changes:

```bash
pylint $(git ls-files '*.py')
```

Please make sure that your code passes the pylint checks before submitting a pull request.

### Commit your changes

We use gitmoji to add emojis to our commit messages. This helps us to quickly identify the purpose of a commit. You can
find the list of available emojis and their meaning [here](https://gitmoji.dev/).

Please follow this convention when writing commit messages:

```
:emoji: Short description of the change (less than 50 characters)
Longer (optional) description of the change (wrap at 72 characters)
```

Please describe your changes in detail in the commit message. This will help us to understand what you have changed and
why.

Also, always use the imperative, present tense: "change" not "changed" nor "changes".

Example:

```
:sparkles: Add data loader for CSV files
```

### Submit a pull request

Push your changes to your forked repository and submit a pull request against the `main` branch of the original
repository.

To push your changes to your forked repository, run the following command:

```bash
git push origin BRANCH_NAME
```

Afterward you can submit a pull request from the GitHub interface.

We require all CI/CD pipelines to pass before merging a pull request. Please make sure that all checks pass and fix
failing checks if necessary.

## Contributing Documentation

If you are interested in contributing to the documentation, please follow these steps:

1. [Fork the repository and clone it](#fork-the-repository)
2. Create a new branch from the `docs` branch by running the following command: `git checkout -b BRANCH_NAME docs`
3. [Add new documentation or update existing documentation](#add-new-documentation-or-update-existing-documentation)
4. [Commit your changes](#commit-your-changes)
5. [Submit a pull request against the `docs` branch](#submit-a-pull-request)

:tada: Congratulations! You have successfully contributed to the documentation of pystreamapi!

### Add new documentation or update existing documentation

The documentation is written in [Markdown](https://www.markdownguide.org/). You can find the documentation in the root
of the branch.

You can access the existing documentation [here](https://pystreamapi.pickwicksoft.org/).
60 changes: 53 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -26,7 +26,7 @@ Now you might be wondering why another library when there are already a few impl
* The implementation achieves 100% test coverage.
* It follows Pythonic principles, resulting in clean and readable code.
* It adds some cool innovative features such as conditions or error handling and an even more declarative look.
* It provides loaders for various data sources such as CSV
* It provides loaders for various data sources such as CSV, JSON, XML and YAML files.

Let's take a look at a small example:

@@ -213,23 +213,69 @@ Stream.concat(Stream.of([1, 2]), Stream.of([3, 4]))

Creates a new Stream from multiple Streams. Order doesn't change.

## Use loaders: Load data from CSV files in just one line
## Use loaders: Load data from CSV, JSON, XML and YAML files in just one line

PyStreamAPI offers a convenient way to load data from CSV files. Like that you can start processing your CSV right away without having to worry about reading and parsing the file.
PyStreamAPI offers a convenient way to load data from CSV, JSON, XML and YAML files. Like that you can start processing your
files right away without having to worry about reading and parsing the files.

You can import the loader with:
You can import the loaders with:

```python
from pystreamapi.loaders import csv
from pystreamapi.loaders import csv, json, xml, yaml
```
Now you can use the loader directly when creating your Stream:
Now you can use the loaders directly when creating your Stream:

For CSV:

```python
Stream.of(csv("data.csv", delimiter=";")) \
.map(lambda x: x.attr1) \
.for_each(print)
```
You can access the attributes of the CSV rows directly like you would with a normal object.

For JSON:
```python
Stream.of(json("data.json")) \
.map(lambda x: x.attr1) \
.for_each(print)
```

You can access the attributes of the data structures directly like you would do with a normal object.

For XML:

In order to use the XML loader, you need to install the optional xml dependency:

```bash
pip install streams.py[xml_loader]
```

Afterward, you can use the XML loader like this:

```python
Stream.of(xml("data.xml")) \
.map(lambda x: x.attr1) \
.for_each(print)
```

The access to the attributes is using a node path syntax. For more details on how to use the node path syntax, please
refer to the [documentation](https://pystreamapi.pickwicksoft.org/reference/data-loaders).

For YAML:

In order to use the YAML loader, you need to install the optional yaml dependency:

```bash
pip install streams.py[yaml_loader]
```

Afterward, you can use the YAML loader like this:

```python
Stream.of(yaml("data.yaml")) \
.map(lambda x: x.attr1) \
.for_each(print)
```

## API Reference
For a more detailed documentation view the docs on GitBook: [PyStreamAPI Docs](https://pystreamapi.pickwicksoft.org/)
Loading