[Alpha pipeline] Write to hub component #140
Conversation
PhilippeMoussalli
requested review from
RobbeSneyders,
GeorgesLorre and
NielsRogge
There was a problem hiding this comment.
Thanks @PhilippeMoussalli!
Now this component isn't really generic though. Would be great if we could automatically define which data to write based on the component spec instead of hardcoded.
| @@ -0,0 +1,57 @@ | |||
| """ | |||
| This component write an images dataset to the hub. | |||
There was a problem hiding this comment.
| This component write an images dataset to the hub. | |
| This component writes an images dataset to the hub. |
I think for this we will need to implement the |
|
We do have access to the component spec as an attribute on the component class. Agree that we'll want to refactor this into a For me it's fine to implement this as a custom component first as well, so it works in the pipeline, and we can do the work to make it reusable later. |
| # login | ||
| huggingface_hub.login(token=hf_token) | ||
|
|
||
| # Create hub |
There was a problem hiding this comment.
| # Create hub | |
| # Create repository |
| # TODO: ADD Arguments for embedding component later on | ||
| # MODEL_ID = "openai/clip-vit-large-patch14" | ||
| # BATCH_SIZE = 10 | ||
| write_to_hub_op = FondantComponentOp( |
There was a problem hiding this comment.
Are you going to include this op in the pipeline?
There was a problem hiding this comment.
yeah indeed, gonna change the simple pipeline to the sd one and then i'll connect all the components together
| @@ -0,0 +1,57 @@ | |||
| """ | |||
| This component write an images dataset to the hub. | |||
There was a problem hiding this comment.
| This component write an images dataset to the hub. | |
| This component write an image dataset to the hub. |
| *, | ||
| dataframe: dd.DataFrame, |
There was a problem hiding this comment.
| *, | |
| dataframe: dd.DataFrame, | |
| dataframe: dd.DataFrame, | |
| *, |
It might be that you'll need to change this to this to run the component successfully
There was a problem hiding this comment.
I think it should be fine since the FondantTransformComponent is currently defined like this
def transform(self, *args, **kwargs) -> dd.DataFrame
There was a problem hiding this comment.
Yes, but it's documented that the dataframe is passed in as a positional argument. We should change this to actually be the case 😛
There was a problem hiding this comment.
ohh 😅 i'll change it, though would opt for something better than documentation for enforcing certain constrains
PR that adds the write to hub component. Right now we're running `to_parquet` method twice, once within the component itself to write to the hub and another time in fondant to write the dataset. We should work on implementing a `DataWriter` class huggingface/huggingface_hub#138. Another thing that we need to configure is how to properly pass secrets huggingface/huggingface_hub#139. Currently the hf token is passed as an argument that is visible in kubeflow to all users who have access to it. Linking also the issue for properly representing submitted images on the hub (right now they are represented as byte strings: huggingface/datasets#5869
PR that adds the write to hub component.
Right now we're running
to_parquetmethod twice, once within the component itself to write to the hub and another time in fondant to write the dataset. We should work on implementing aDataWriterclass huggingface/huggingface_hub#138.Another thing that we need to configure is how to properly pass secrets huggingface/huggingface_hub#139. Currently the hf token is passed as an argument that is visible in kubeflow to all users who have access to it.
Linking also the issue for properly representing submitted images on the hub (right now they are represented as byte strings:
huggingface/datasets#5869