Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache scraper results #15

Open
chimon2000 opened this issue Feb 28, 2019 · 3 comments
Open

Cache scraper results #15

chimon2000 opened this issue Feb 28, 2019 · 3 comments
Labels
enhancement New feature or request must have A feature that needs to be implemented

Comments

@chimon2000
Copy link
Collaborator

chimon2000 commented Feb 28, 2019

Is your feature request related to a problem? Please describe.
Currently, the scraper results are stored in a JSON file. This will not scale well, so we should choose a better caching solution.

Describe the solution you'd like
Ideally, we would store the results in Firestore. This gets rid of the need to maintain and extend an API.

A secondary option is caching the results in an appropriate caching solution (either Memcache or Redis)

Results that are more than 30 days in the past should be expired.

Describe alternatives you've considered
n/a

Additional context
n/a

@chimon2000 chimon2000 added must have A feature that needs to be implemented enhancement New feature or request labels Feb 28, 2019
@chimon2000 chimon2000 pinned this issue Feb 28, 2019
@AndrewNatoli
Copy link
Member

Suggestion: Since we're currently doing a full pull when we run the spiders we might as well wipe and replace the full cache on run instead of worrying on managing the TTL of data for now

@chimon2000
Copy link
Collaborator Author

chimon2000 commented Apr 11, 2019

I actually decided that maybe we are overthinking this. Let's just store the results in Firestore, scrap the idea for an API, and have the app pull directly from the Firestore.

@chimon2000
Copy link
Collaborator Author

Here is the current structure of Community events

type CommunityEvent = {
id: number
title: string
description: string
date: string
tags: string[]
locations: EventLocation[]
learnMoreUrl: string
time: string
}
type EventLocation = {
name: string
locationRoom: string
locationUrl: string
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request must have A feature that needs to be implemented
Projects
None yet
Development

No branches or pull requests

2 participants