Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local Docker #199

Open
TonyBrobston opened this issue Jan 3, 2024 · 22 comments
Open

Local Docker #199

TonyBrobston opened this issue Jan 3, 2024 · 22 comments

Comments

@TonyBrobston
Copy link

Hey Gleb, thanks for your work on this project.

I saw in your blog post here: https://glebbahmutov.com/blog/cypress-parallel-free/ that this project could be used locally using Docker containers.

I spent the better part of a day playing around to see what I could get to work, but I haven't come up with a complete solution, but have had some success.

Do you have a proposed solution?
@bahmutov
Copy link
Owner

bahmutov commented Jan 3, 2024 via email

@TonyBrobston
Copy link
Author

To run Docker containers locally or to set up split indices?

To run docker containers locally and split up cypress tests. We are currently running these tests in CircleCI with parallelism: 9, the hope is to do the same thing, but locally with docker.

I'm not sure I follow the second part of your question, are you just referring to splitting the tests?

@bahmutov
Copy link
Owner

bahmutov commented Jan 3, 2024 via email

@TonyBrobston
Copy link
Author

I will see if I have time to set up an example, but I might have to make it a lesson in my cypress-split course since it is a time investment

That works for me.

With fast-feedback being super important to development, having these tests run in parallel locally seems like the next step in the progression.

@bahmutov
Copy link
Owner

bahmutov commented Jan 3, 2024 via email

@TonyBrobston
Copy link
Author

TonyBrobston commented Jan 3, 2024 •
edited
Loading

I wonder if it would work - the thing about local docker containers is that the local machine often struggles and the tests slow down a lot

I believe it will.

This isn't quite comparing apples to apples. But about 2 years ago I did some load testing with Locust and used their scale option (https://docs.locust.io/en/stable/running-in-docker.html#docker-compose). IIRC when I ran a single container, I could not produce as many requests as I needed in a given amount of time. When I switched to scaling, I was able to produce significantly more, but I eventually hit the limits of the hardware I was on (an Intel Mac). I had my coworker with better hardware (an M1 Mac) pull down the repo and run the same Locust scaling setup with more workers and we were able to produce even more requests, which ended up being sufficient.

This may be comparing apples to oranges, so correct me if I'm wrong. However, to me it seems like something worth investigating.

@TonyBrobston
Copy link
Author

TonyBrobston commented Jan 3, 2024 •
edited
Loading

Maybe we can come up with a POC solution and see what kind of results it produces.

Earlier I was playing around with the split and splitIndex that you mentioned in "Other CIs": https://github.com/bahmutov/cypress-split/tree/main?tab=readme-ov-file#other-cis

It seems that using docker-compose up cypress --scale cypress=2 in combination with the HOSTNAME environment variable and calling the docker engine api, we could get the information needed to populate split and splitIndex.

For split we could count the number of containers running using the same service name. To do this, we could call: curl --unix-socket /var/run/docker.sock http://localhost/v1.41/containers/json which returns the list of running containers. Using the container id, which is stored in the HOSTNAME environment variable, we can get its container, and inside the container object we can get the service name. From there we can count the number of running containers that are using the same service name.

For splitIndex we could take the service name we retrieved before, filter down the list of running containers with the same service name. Then return the index of the container whose id matches the HOSTNAME.

I'm just unsure of where to put this logic and then the best way to pass these values to the cypress command.

Here is the docker-compose I was experimenting with:

version: "3.4"

services:
  cypress:
    image: cypress/included
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    entrypoint: sh -c "apt-get update && apt-get install -y curl && env && curl --unix-socket /var/run/docker.sock http://localhost/v1.41/containers/json && sleep 1000"

Thoughts?

@TonyBrobston
Copy link
Author

After thinking for a bit. I'll write a quick JavaScript package to do what I've described above and wrap this package.

I'll do two runs locally, one as all tests in order with no parallelization, and another with n number of threads where n is the optimal amount. I imagine n will be 8-15 in my case with about 100 tests across 15 files. Then I'll report back my findings.

My hypothesis is that on machines with good specs, I'll see no slow down at 15 containers. But, maybe not.

@TonyBrobston
Copy link
Author

TonyBrobston commented Jan 4, 2024 •
edited
Loading

Here's where I ended up.

https://github.com/TonyBrobston/cypress-split-local-docker

I didn't want to publish this to npm, so I just installed from github: npm install -D git+ssh://[email protected]/TonyBrobston/cypress-split-local-docker.git

version: "3.4"

services:
  cypress:
    image: cypress/included
    environment:
      - LOCAL=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - .:/home
    working_dir: /home
    entrypoint: npx cypress run
image

It is getting late. I wasn't able to do the two local runs yet as I'm having an error that I think is related to running our React application on the host machine and Cypress in docker. I'll hopefully get that worked out in the morning and then report my findings.

@TonyBrobston
Copy link
Author

@bahmutov

I have finally overcame an issue I was having with running everything in docker.

I'm not sure if I'm using the "Other CIs" option incorrectly or if there is a bug. It seems like the output at the beginning is correct, but then later it seems to be ignored.
image
image
image

At the beginning it says that cypress/e2e/assign-jobs.cy.js will run in chunk 1 of 2, but then it tries to run on both workers.

I tried two different approaches to setting split and splitIndex, both seemed to produce the same result:

  1. https://github.com/TonyBrobston/cypress-split-local-docker/blob/e55ceb078f5db8d82da7db31a5a9f8f0543f8832/src/index.js#L15-L20
  2. https://github.com/TonyBrobston/cypress-split-local-docker/blob/12b6ac3d4e947d90f0d4399e118880ffb911fc68/src/index.js#L15-L22

@TonyBrobston
Copy link
Author

Actually... I think I see a bug on my side.

@TonyBrobston
Copy link
Author

Actually, maybe not. I thought one of the two parameters (split or splitIndex) was incorrect, but I logged the output and they look correct.

image

@TonyBrobston
Copy link
Author

TonyBrobston commented Jan 8, 2024 •
edited
Loading

After looking at this for awhile, it appears I just needed to await in setupNodeEvents, since I am doing a network call that is awaited inside cypress-split-local-docker.

async setupNodeEvents(on, config) {
  await require('cypress-split-local-docker')(on, config);
  return config;
},

@TonyBrobston
Copy link
Author

@bahmutov I can finally report some results.

Running Cypress tests synchronously (on my machine outside of docker), it took 4:33 to run 101 tests. Running Cypress tests in docker with 4 instances (docker-compose up cypress --scale cypress=4), it took 1:27 to run 101 tests.

image

When I went up to 5 instances I started getting 429's from our api. Since we also have this running in docker, I hope to work around that issue and continue to increase the number of instances until things start to crash. I did attempt 9 instances and did get this error:
image

I think it is pretty clear that, on decent hardware, this solution is viable.

Thoughts?

@bahmutov
Copy link
Owner

I love it. Is there a repo with a simple setup showing your compose file?

@TonyBrobston
Copy link
Author

I love it. Is there a repo with a simple setup showing your compose file?

Thanks. I'm glad to hear that. If it wasn't for this repo and some suggestions from my coworkers I don't think I could have figured it out.

I'll update my readme tomorrow with instructions, but if I'm thinking correctly they are also in this thread. Though it is probably more concise if I write the instructions start to finish.

Would you be open to adding this code to your project? Seems like it fits here and... I really don't want to maintain it if at all possible.

@bahmutov
Copy link
Owner

bahmutov commented Jan 12, 2024 via email

@TonyBrobston
Copy link
Author

TonyBrobston commented Jan 16, 2024 •
edited
Loading

@bahmutov I didn't get around to this until this morning; I also realized I'm an air-head and accidently made this repo private, it is now public. I got the readme updated.

The repo is very short and concise, so hopefully it is straight forward.
https://github.com/TonyBrobston/cypress-split-local-docker

Let me know if you have any questions, concerns, ideas, etc.

@TonyBrobston
Copy link
Author

@bahmutov Have you had a chance to take a look?

@bahmutov
Copy link
Owner

bahmutov commented Jan 24, 2024 via email

@euddin1
Copy link

euddin1 commented May 17, 2024

I'd be very interested in this. And if it ends up being a lesson in the course I'd pay for it. I'm going to look through the plugin you created @TonyBrobston to see if I can get it to work but it would be great if the cypressSplit plugin could do it on its own.

@TonyBrobston
Copy link
Author

I'd be very interested in this. And if it ends up being a lesson in the course I'd pay for it. I'm going to look through the plugin you created @TonyBrobston to see if I can get it to work but it would be great if the cypressSplit plugin could do it on its own.

I agree.

Let me know if you run into any issues. Hopefully it is pretty straight forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bahmutov @TonyBrobston @euddin1