Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: API for uploading a sway project #19

Merged
merged 12 commits into from
Dec 5, 2024
Merged

Conversation

sdankel
Copy link
Member

@sdankel sdankel commented Sep 24, 2024

Closes #18

This PR adds a new API, upload_project, that accepts a .tgz file containing a sway project and does the following:

  1. Strips out irrelevant files
  2. Attempts to build the project
  3. Uploads the ABI file IPFS and stores the hashes
  4. Generates a bytecode ID from the compiled binary and stores it in a new postgres table
  5. Creates a new .tgz from the cleaned up source code and uploads it to IPFS
  6. Returns an upload ID, which can be used to publish metadata about the uploaded project (coming in a future PR)

Notes:

  • Because of the limitations of pinata and the pinata/IPFS rust SDKs, the source code is stored in IPFS as a .tgz file rather than as a uncompressed directory.
  • In order to build the sway project with the correct version of forc, we pre-install the latest 20 versions of forc on the docker image. As a fallback, the service uses cargo binstall to fetch the version of forc if it isn't pre-installed. This means the API can take a while to run if the version requested is not already installed.
  • I added one integration test for the happy path, but I didn't want to add more with different versions because the test takes a while to run. Happy to add more tests as needed.

Copy link

vercel bot commented Sep 24, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
forc-pub ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 3, 2024 7:19am

@sdankel sdankel changed the title feat: API for file uploading feat: API for uploading a sway project Sep 25, 2024
@sdankel sdankel requested a review from a team November 21, 2024 17:32
@sdankel sdankel marked this pull request as ready for review November 21, 2024 17:32
src/api/mod.rs Outdated Show resolved Hide resolved
Copy link
Member

@JoshuaBatty JoshuaBatty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work! just a couple of nits

src/main.rs Outdated Show resolved Hide resolved
src/models.rs Show resolved Hide resolved
src/pinata/mod.rs Show resolved Hide resolved
src/upload.rs Show resolved Hide resolved
src/upload.rs Outdated Show resolved Hide resolved
@JoshuaBatty JoshuaBatty requested a review from a team November 22, 2024 01:37
src/pinata/mod.rs Outdated Show resolved Hide resolved
src/main.rs Outdated Show resolved Hide resolved
src/upload.rs Outdated Show resolved Hide resolved
}

/// Installs the given version of forc at the specific root path using cargo-binstall.
pub fn install_forc_at_path(forc_version: &str, forc_path: &Path) -> Result<(), UploadError> {
Copy link

@zees-dev zees-dev Nov 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using the CLI; i'd reccomend using library/SDK functions to perform uploading operations.
They would be much lighter and ephemeral as one would not need to have the binary installed.
^ could be a future PR; just something to keep in mind

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you saying to use forc-pkg to build the sway project rather than using the forc CLI? Or are you suggesting using a library to install forc? If you know of libraries that can do this I'd be interested, I looked around but couldn't find one.

Copy link

@zees-dev zees-dev Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea i was saying we shouldn't rely on the user having forc-binaries installed; implying requiring external deps - outside this project.
Note: Could be a future PR though as an improvement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be a little lost here but isn't this a server that we are running? What users are we referring to exactly?

My understanding was that: we run this service, once a user want's to actually publish a project, forc upload (or whatever the cli command will be) will make an API call into this service uploading the project.

Then the server will get the project, and extract the bytecode identifier from it (bytecode identifier in this context would basically be an id generated from the existing bincode that "generalizes" the project, meaning that the effect of different initial configurable values etc are removed to get an id that describes only the "core" of the contract)

Copy link
Member Author

@sdankel sdankel Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kayagokalp has it right. This isn't expecting a user to have forc binary installed, it's installing it on the server (running in docker in the cloud) and using it to build the sway project that it received via the upload API.

We wouldn't be able to use forc-pkg since we have to support any version of forc.

.env Outdated Show resolved Hide resolved
@alfiedotwtf
Copy link

Nice! Happy to approve once updated and you're happy for another review.

src/main.rs Show resolved Hide resolved

// Clean up the temporary directory.
fs::remove_dir_all(upload_dir).map_err(|_| ApiError::Upload(UploadError::SaveFile))?;
tmp_dir
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whole point of tempfile or tempdir is not requiring explicit cleanup since the Drop trait for this should do this when the variable goes out of scope (I think)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs say to call tmpdir.close()

uses: taiki-e/install-action@cargo-binstall
- name: Install forc for tests
run: |
cargo binstall --no-confirm --root tests/fixtures/forc-0.65.0 [email protected]
Copy link

@alfiedotwtf alfiedotwtf Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the version string be dynamic so it doesn't have to be touched every update?

Just a question, happy for this to be merged though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an argument for it but usually we don't want tests to start failing because of a release in another repo.

Copy link
Member

@kayagokalp kayagokalp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me, I got couple final things in my hand after that I'll be switching to getting metadata stuff ready. Excited to get that and this working together to see forc package registry coming alive!

Comment on lines +52 to +65
/// A mock implementation of the PinataClient trait for testing.
#[cfg(test)]
pub struct MockPinataClient;

#[cfg(test)]
impl PinataClient for MockPinataClient {
async fn new() -> Result<Self, UploadError> {
Ok(MockPinataClient)
}

async fn upload_file_to_ipfs(&self, _path: &Path) -> Result<String, UploadError> {
Ok("ABC123".to_string())
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really not important but these two can be moved to a mod test if that sounds good to you (to follow a more conventional style), totally ok if we leave them as is though!


/// Handles the project upload process by unpacking the tarball, compiling the project, copying the
/// necessary files to a new directory, and storing the source code tarball and ABI file in IPFS.
/// Returns a NewUpload struct with the necessary information to store in the database.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Returns a NewUpload struct with the necessary information to store in the database.
/// Returns a `NewUpload` struct with the necessary information to store in the database.

}

/// Installs the given version of forc at the specific root path using cargo-binstall.
pub fn install_forc_at_path(forc_version: &str, forc_path: &Path) -> Result<(), UploadError> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be a little lost here but isn't this a server that we are running? What users are we referring to exactly?

My understanding was that: we run this service, once a user want's to actually publish a project, forc upload (or whatever the cli command will be) will make an API call into this service uploading the project.

Then the server will get the project, and extract the bytecode identifier from it (bytecode identifier in this context would basically be an id generated from the existing bincode that "generalizes" the project, meaning that the effect of different initial configurable values etc are removed to get an id that describes only the "core" of the contract)

Copy link
Member

@JoshuaBatty JoshuaBatty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really, great stuff. Feel free to address some of the remaining small nits in a follow up PR if you just want to get this over the line.

Copy link

@zees-dev zees-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks great! 🚀

@sdankel sdankel merged commit 7b4838b into master Dec 5, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upload project API
6 participants