Need a mechanism for deploying edX courses

[usernamenumber]

We could do this manually for the initial test deployment if needed, but ideally we should have:

• Scripts that make it easy to adapt an edX course for local hosting (download videos, etc)
  • Related: what's the best way to do the actual hosting? Would we benefit from having some kind of streaming server, or just store videos in a known location and embed them with html5?
• Scripts that make it easy to import a course into the local instance of edX LMS
  • Maybe this already exists?

[devalih]

Importing course feature already there by edx:

cd /var/tmp
  git clone https://github.com/edx/edx-demo-course.git
  cd /edx/app/edxapp/edx-platform
  sudo -u www-data /edx/bin/python.edxapp ./manage.py cms --settings=aws import /edx/var/edxapp/data  /var/tmp/edx-demo-course

For video streaming, We should include a new role "media server" to host static files and videos under nginx, this should be faster than using django to deliver that.

I already have a script to download videos from youtube and update the MongoDB to point to the new location, I'll clean that and add a pull request soon.

[devalih]

Some questions:

Does --settings=aws has to do with the AWS service of Amazon? Please
explain its use.

edX production setup is so dependent on some amazon services such as S3, and yes aws settings has some of this configurations.
You can use other env files as you desire, check here

Where exactly is stored the Course data? What if we can store it on local
hard drive?

edX stores course data on mongodb, so the import command imports the course from the local disk to mongo. or in the below case from github to local disk to mongodb.

Is it possible to have a more sophisticate course to test features like
"Liner Algebra"? May a more sophisticated demo course be developed so we
can test a more complete set of EDX features?

Yupe of course, you can use edX-Insider course (developed by master Piotr) it uses much more features than the demo course.

I know is not difficult on EDX Studio to create a course, but is it easy to
include these sophisticated features?

You can fork it and add whatever you want to test, and yes I agree some of the advance problem types are somehow hard to create for the average TA, but it shouldn't be a problem to anybody with a some programing skills.

Thanks for your answers.

Thank you for your questions :)

[iLearner]

I modified the EDX installation (Lubuntu 13) so it creates the SD card
which includes, besides Lubuntu 13, EDX also so you don't have to install
it. I only replaced the folder tree that contains the OS files with the one
that has a complete installation of EDX. This requires that we have this
compressed "tar.gz" folder tree, which has 2 GB, some where in the internet
to be used by the script. I am currently uploading it to my Google Drive
account, but I don't know if this is the right way. Please give
suggestions. I believe this installation is complete. The "Demo" and EDX
Insider" courses seems to work find, also from the EDX studio, but i still
need a way to confirm if "Linear Algebra" and the "Apparmor" Module are
working.

With this Method you can create the SD Card and also an Image of it, as we
have already both types of scripts. The size of the full Image, otherwise,
takes 5 GB and is more tricky to make because of the empty space it has to
have.

We can use this to do tests while we continue working on the Ansible-Docker
Solution.

I also will try to have the Ubuntu 14 Installation version totally complete
so we have two choices.

On Tue, Dec 23, 2014 at 1:17 PM, Ali [email protected] wrote:

Some questions:

Does --settings=aws has to do with the AWS service of Amazon? Please
explain its use.
edX production setup is so dependent on some amazon services such as S3,
and yes aws settings has some of this configurations.
You can use other env files as you desire, check here
https://github.com/edx/edx-platform/tree/master/lms/envs

Where exactly is stored the Course data? What if we can store it on local
hard drive?
edX stores course data on mongodb, so the import command imports the
course from the local disk to mongo. or in the below case from github to
local disk to mongodb.

Is it possible to have a more sophisticate course to test features like
"Liner Algebra"? May a more sophisticated demo course be developed so we
can test a more complete set of EDX features?
Yupe of course, you can use edX-Insider
https://github.com/pmitros/edX-Insider course (developed by master
Piotr) it uses much more features than the demo course.

I know is not difficult on EDX Studio to create a course, but is it easy
to
include these sophisticated features?
You can fork it and add whatever you want to test, and yes I agree some of
the advance problem types are somehow hard to create for the average TA,
but it shouldn't be a problem to anybody with a some programing skills.

Thanks for your answers.
Thank you for your questions :)

—
Reply to this email directly or view it on GitHub
#30 (comment).

[usernamenumber]

Thanks to both of you for the work you've done on this!

@iLearner, when the image finishes uploading, please post the link here. Also, does it include the ansible playbooks for edX, or just the services they install?

[iLearner]

Here is the link:
https://drive.google.com/file/d/0B2Ncx5bTTuA8UVFZRF9majdfS3M/view?usp=sharing
On the SD script you have to use that file instead of
"rootfs-part2.tar.gz". That's it.
I tried to make it downloadable with Ubuntu "wget", but seems that it is
not possible with Google Drive.
If we can find some place where it can be done like that, it may be would
be good.

It includes the Root folder tree as it is when the EDX installation
completes by following my EDX Lubuntu 13 Installation procedure on
https://github.com/iLearner/x2go/tree/Lubuntu-13

On Wed, Dec 24, 2014 at 1:02 PM, Brad Smith [email protected]
wrote:

Thanks to both of you for the work you've done on this!

@iLearner https://github.com/iLearner, when the image finishes
uploading, please post the link here. Also, does it include the ansible
playbooks for edX, or just the services they install?

—
Reply to this email directly or view it on GitHub
#30 (comment).

[usernamenumber]

I am downloading this (internet is not very good where I am right now), but won't be able to test it until I can go out and get a new SD card (brought one with me, but it turns out it's bad).

Once I have it downloaded, I can at least put the file up in a temporary location that can be accessed with wget, and I'll talk to the Tunapanda folks about hosting it at tunapanda.org.

[usernamenumber]

I am back home now and have a new SD card. I'm extracting the tarball, but have an issue/question:

I was expecting a filesystem image I would then dd onto the card, but this appears to be a regular file archive. Do I need to edit the scripts in your iLearner/x2go repo to use this archive and then do execute sd with them?

[iLearner]

It is not an image because in that case it will be at least 5 GB big which is the size of the uncompressed root folder tree that you get when completing the EDX installation on Ubuntu 13. Instead It contains only the compressed Root folder tree as a "tar.gz" file. So to make it work use this file instead of the "rootfs-part2.tar.gz" file on the "sd" script to create the SD card (You can take it from https://github.com/iLearner/x2go/tree/Lubuntu-13) , or on the script we modified later from there to get directly the SD Card Image.. The late would be at least 10 GB as we did the script and would be to big to manipulate.
After replacing the text you can do "python execute sd /dev/sd?". don't forget to have also the "execute" and "sd_card_partitions.out" files in the same folder.

[usernamenumber]

I tried the following:

1. Clone the Lubuntu-13 branch of iLeaner/x2go/
2. Comment out the line in sd that wgets the rootfs-part2.tar.gz file
3. Download the archive linked above and save it as rootfs-part2.tar.gz in the x2go dir
4. Run python execute sd /dev/sdb

After unpacking the archive, execute fails with this:

sync
OK!

mkdir /tmp/sdd2/home/linaro/EDX
mkdir: cannot create directory `/tmp/sdd2/home/linaro/EDX': File exists
ERROR!

Nothing was mounted there before, and the SD card was empty, so I'm not sure what's happening. It's getting late over here, so I'll pick this up again tomorrow. Just posting now in case you see an obvious fix.

[iLearner]

Sorry for that, I forgot that for this it is not needed, because it is
already created.

# Copy EDX Required Installation Files to /home/linaro/EDX on SD

Card

mkdir /tmp/sdd2/home/linaro/EDX

cp README.md /tmp/sdd2/home/linaro/EDX/README.md

cp execute /tmp/sdd2/home/linaro/EDX/execute

cp update /tmp/sdd2/home/linaro/EDX/update

cp edxPRE /tmp/sdd2/home/linaro/EDX/edxPRE

cp fixes /tmp/sdd2/home/linaro/EDX/fixes

cp edx /tmp/sdd2/home/linaro/EDX/edx

cp text_replacements.xml

/tmp/sdd2/home/linaro/EDX/text_replacements.xml

cp mongo_tasks_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_tasks_main.yml

cp mongo_defaults_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_defaults_main.yml

but the SD is already created because you only missed this section
(Unmounting and removing the SD directories and eject):

Unmount SD Card Partitions

sudo umount /tmp/sdd1
sudo umount /tmp/sdd2
sudo rmdir /tmp/sdd1 /tmp/sdd2

Make SD Card Safe to remove (Eject)

sudo eject ${card}

On Sat, Dec 27, 2014 at 10:25 PM, Brad Smith [email protected]
wrote:

I tried the following:

1. Checkout the Lubuntu-13 branch of iLeaner/x2go/
2. Comment out the line in sd that wgets the rootfs-part2.tar.gz file
3. Download the archive linked above and save it as rootfs-part2.tar.gz
   in the x2go dir
4. Run python execute sd /dev/sdb

After unpacking the archive, execute fails with this:

sync
OK!

mkdir /tmp/sdd2/home/linaro/EDX
mkdir: cannot create directory `/tmp/sdd2/home/linaro/EDX': File exists
ERROR!

Nothing was mounted there before, and the SD card was empty, so I'm not
sure what's happening. It's getting late over here, so I'll pick this up
again tomorrow. Just posting now in case you see an obvious fix.

—
Reply to this email directly or view it on GitHub
#30 (comment).

[iLearner]

I sent too fast the last email. Just to clarify, for future use of the SD
script you need to delete or comment the EDX Installation files' section,
because, EDX is already Installed in the Root folder tree provided, and I
left those files there.

Any way, the SD card should be create completely by now and you can try to
load it on Cubietruck.
In my experience it is always good idea to eject the SD card so to take it
out properly. Some times when you don't do it properly, it may fail to load
on Cubietruck and you need to redo the SD creation process.

On Sun, Dec 28, 2014 at 12:17 AM, Arturo Samanez [email protected] wrote:

Sorry for that, I forgot that for this it is not needed, because it is
already created.

# Copy EDX Required Installation Files to /home/linaro/EDX on SD

Card

mkdir /tmp/sdd2/home/linaro/EDX

cp README.md /tmp/sdd2/home/linaro/EDX/README.md

cp execute /tmp/sdd2/home/linaro/EDX/execute

cp update /tmp/sdd2/home/linaro/EDX/update

cp edxPRE /tmp/sdd2/home/linaro/EDX/edxPRE

cp fixes /tmp/sdd2/home/linaro/EDX/fixes

cp edx /tmp/sdd2/home/linaro/EDX/edx

cp text_replacements.xml

/tmp/sdd2/home/linaro/EDX/text_replacements.xml

cp mongo_tasks_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_tasks_main.yml

cp mongo_defaults_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_defaults_main.yml

but the SD is already created because you only missed this section
(Unmounting and removing the SD directories and eject):

Unmount SD Card Partitions

sudo umount /tmp/sdd1
sudo umount /tmp/sdd2
sudo rmdir /tmp/sdd1 /tmp/sdd2

Make SD Card Safe to remove (Eject)

sudo eject ${card}

On Sat, Dec 27, 2014 at 10:25 PM, Brad Smith [email protected]
wrote:

I tried the following:

1. Checkout the Lubuntu-13 branch of iLeaner/x2go/
2. Comment out the line in sd that wgets the rootfs-part2.tar.gz file
3. Download the archive linked above and save it as
   rootfs-part2.tar.gz in the x2go dir
4. Run python execute sd /dev/sdb

After unpacking the archive, execute fails with this:

sync
OK!

mkdir /tmp/sdd2/home/linaro/EDX
mkdir: cannot create directory `/tmp/sdd2/home/linaro/EDX': File exists
ERROR!

Nothing was mounted there before, and the SD card was empty, so I'm not
sure what's happening. It's getting late over here, so I'll pick this up
again tomorrow. Just posting now in case you see an obvious fix.

—
Reply to this email directly or view it on GitHub
#30 (comment).

[usernamenumber]

During a recent meeting, @pmitros pointed us to some freely available edX courses provided by MIT:

• https://github.com/mitocw/6-00sc-spring-2011
• https://github.com/mitocw/18-02sc-fall-2010.edx

And here are a couple of edX courses on more basic subjects, which they made by converting Moodle courses from moodleshare using their moodle2edx tool. These courses might not be as well-designed as the edX courses, but if the script works reliably, they might provide an easy way to include a broader range of topics with the initial deployment by converting other courses from moodleshare.

• https://github.com/mitocw/content-ocw-intro-to-stats
• https://github.com/mitocw/content-ocw-explore-engineering

@devalih: Earlier you said that it would be pretty easy to put together a script for this. Do you think you could put something together for testing some time soon and use it to import these courses? One request if you do: please have your script download the videos to /usr/local/tunapanda/data/video. I've got a related feature I'm thinking about that could do some cool things with that.

[usernamenumber]

And here's an electronics course that may be a better fit than either of the first two MIT courses linked above. My understanding from this would be a good one to prioritize: http://groups.csail.mit.edu/mac/users/gjs/6.002x/

[usernamenumber]

Ping @devalih re the video migration script mentioned in #30 (comment)