-
Notifications
You must be signed in to change notification settings - Fork 2
ACT Voyager Grant Proposal
Act, as we know it, has been build more than a decade ago for Apache 1 and mod-perl 1.3 architecture. And it has served the Perl Community very well over all those years and hundreds of events.
But over the last year we have suffered from major outages. And since the operating sytem it self being used to run on, has passed it LTS, more recent compilers being packed, it has become extremely difficult (if not impossible) to rebuild Act from scratch.
Also, Act as it is now, is no longer maintained in a pro-active manner, due to it's archaic design - and people having other priorities.
Lastly, due to the flexibility off how it has been designed it comes with such a complexity, that it is hard and difficult for current organisers to deliver new websites for each and every conference or workshop they want to promote. Of course, this also relates to the fact that most of the Perl developers are not the best frontend designers.
Reasons enough to stop using Act, as a toolkit, it has become rusty and unfit for the job. It has come to the point that Act should be to REST.
The future is quite clear. Act requires a giant leap forward to keep up with the demands of today users. Visitors that come to the website should experience that Perl, although a language with a longstanding history, is still keeping up and pushing the bar to a higher level for competing alternatives. The user experience should be up to modern expectations. Therefor Act Voyager will include a revamped frontend. And hopefully will make it more likely to attract new people to the Community, rather then putting people off on their first impression (as we know from Perl Monks).
Organisers should be enabled to quickly and easaly set up a new event. That should be as easy as setting a date, selecting a frontend theme and additional styling. The admin part of Act would be decoupled from the public website. This makes it much easier to implement, without messing with the public website interface. This admin part would also need new tools that connect with Social Media and allow simple things as sending emails. Previous organisers will recognise all the hassle to get it up and running and most would wish it was more easy to communicate with attendees and prospective Community members.
A 'must have' for working on anything related to Act, more specific, anything related to the Legacy code base.
This YAML specification of an API will describe the interface. Rather than just writing code and deliver something that would look okay, it's important to have a good definition of what the API looks like - much like Test Driven Development.
Based on the above Specification, there will an API server that talks to the Legacy Act databases.
As a side product, mostly for testing purposes, there will be a simple command line interface. Extremely useful for admins and people that dislike modern web.
A Dancer2 application that will provide the HTML, JavaScript, and CSS needed to access a Perl Event web page.
Dancer2::Plugin::HTTP::Auth::Extensible Dancer2::Plugin::HTTP::Caching Dancer2::Plugin::HTTP::ConditionalRequest Dancer2::Plugin::HTTP::ContentNegotiation LPW::UserAgent::Caching HTTP::Caching
Since it's a nightmare to build Act from scratch - DO NOT TRY THIS AT HOME - it comes with Apache 1.3.x and mod-perl 1.31. It also includes a recent checkout of the Git repository and it's dependancies.
This will benefit organisers to work on their website and test locally before committing and merging with the 'Act-Conferences' repository.
It will also help developers that desire to fix bugs in the legacy codebase of Act... or add new features. This part has almost finished. I have gone through dozens and dozens off iterations to get it working. Luckily I had a back up from a previous attempt from some years ago. It only needs some nice script to make it easy for an organiser to add a conference, linked to GitHub.
There are many tools available around the Oen API specification that will help backend developers to test their implementations. There are also tools available to run a API simulation. And many more... The Specification will be a living document and contributions are welcome. But whatever the specification will be, that will be what client and server will to agree on. This will open up the way developers can work on separate products. With a proper Specification, it is possible to built different clients (like iOS applications) and servers.
The specification can be just one big file and will soon become messy. This year I started out writing the specification, split over several files. This will make it much more easy to maintain. The current setup passes the OAS validation.
More endpoints and objects will need to be added. But I also hope that people interested in the project will comment on the repo and make pull-requests.
Most likely, this will be build on top of Dancer2, using a few plugins that deal with the correct handling of HTTP request headers in requests and responses.
This will be a minimal viable product.
Themes used for this server should obtain their data through the REST api. And since it has a strict separation between data and presentation, actually anyone could build their own server that talks to the API ... when following the Specification.
The server will also include a 'page' that enable a user to manage their own settings and make payments.
There are some modules I already published on CPAN, but need a little more after care. Some lack proper tests, some actually fail after changes in Dancer2.
I've written those modules, because I felt they were needed to make this entire project succeed. But the Perl Dancer developers have a good level of minimum requirements before being acknowledged and accepted as real Dancer2 plugins.
LWP::UserAgent::Caching, and it's dependancy HTTP::Caching, require a bit more development. So far, it's the only RFC compliant library.
The module will be needed in the CLI client. The REST api will only serve 'simple' (JSON) objects. And because off that, they are cacheable. But there was no UserAgent that was capable of handling HTTP requests or responses as it was defined by the RFC's.
*create script to add a conference and checkout wit GitHub *save as a Vagrant Box *write documentation for organisers and developers
- add specification for user
- add specification for talk
- add specification for confernce
- add specification for attendee
- add specification for ...
probably quite some rewrite, but there is quite some willingness to support that work from the Dancer Core developers
- add tests and anything else that would enhance the kwalitee
- add tests and anything else that would enhance the kwalitee
- add tests and anything else that would enhance the kwalitee
- Responses should have Age header
- successful Delete request should delete the cached data
This requires a little chat with some respectable Perl developers. But basically I want a true separation of the business object and how it has been retrieved or stored or updated.
This makes it possible to use the same object on the CLI and on the server, with the only difference that the former talks to the REST api and the latter to the database.
- create simple route
- add authentication and authorization
- add more routes as described in the Open API specification
- figure out how to easy build a CLI from a Swagger file
- Tina Muller has done some tremendous work on that
- get it working for a user object
- add more command options
as the Specification is becoming more complete, and the API get more endpoints, the CLI will grow with it.
This requires some more investigation, but I have very strong connection with some very reputable front end developer that knows how to build a front end.
But it basically comes down - when using Angular2 with Bootstrap 4 to just go along with the growth of the API.
The project is quite big, but within the first few weeks, it will at least have made good progress on the first two deliverables.
The hard part will be to get an initial 'prototype' of the REST api working. I expect that to have finished within two months after having the Vagrant Box up and running.
The CLI will follow probably another month later.
From there on, it's a matter of adding more endpoints and command options as the Swagger files gets updated.
A first release of the web-server will probably follow four to six months after the CLI. Since the size of the project, it would be about a year before the MVP will be ready.
Well, a minimal viable product, a working theme-based website. It should at least cover 80% of the common user stories for a general conference attendee. This includes a personal web-page to edit and manage settings. For the other admin tasks, I would stick to a proper CLI for now.
I have initially started work on this project some years ago and am very eager to continue work on this.
I've done a lot of research on REST api's and given some presentations on that.
In the last three years I've also organised three Perl Workshops in the Netherlands - and was the main organiser of The Perl Conference in Amsterdam.
This has given me a lot of insight in how Act has been designed and what the difficulties are with the current legacy version.
A year ago I did give a presentation during the Perl Dancer conference. This was a summary of my work done so far and how all the puzzle pieces would fit together to build Act-Voyager.
Compared to three years ago, I've grown much more in experience as a Perl Developer. This new application is much better than the previous and based on personal experience working on on the project intermittently. While organising TPCiA, I was facing quite some difficulties that caused delay in further development of Act Voyager. Now that has finished, time has come available.
I endorse this proposal. I know that when we first started to use ACT a dozen years ago, it was a real step forward for the Perl community as it relieved conference organizers of one particular burden. It was also beneficial for conference attendees; I could maintain a single account year after years.
Alas, more recently this has not been the case. We had outages at YAPC::NA in 2015 or 2016. We had outages at DCBPW in 2016. I was therefore not surprised when the TPC::NA organizers did not use ACT in Virginia this year. But that meant that the conference organizers had to re-invent certain wheels.
Consequently, I think that this project needs to start with a fact-finding phase. What exactly went wrong during those outages? That information should be translated into stories that the ACT-rewrite needs to address before it can be used in live production.
I'm sorry to say that I don't think it makes sense to continue to work on ACT (for anybody..) It would make much more sense to me to start a completely new product, and migrate the old data into the new system.
But here are some more concrete questions regarding this grant:
- We all agree that the current ACT codebase (and DB!) is horrible (because it's old - it was state of the art when it was written), and some of the main features (eg localisation) have much better solutions now than back in 2004(?). So I don't think it is a good idea to reuse the old database schema. If we want the data, we could export it in a new, saner schema.
- There is already a rewrite (also done using Dancer I think) by Racke / the Dancer Conference people. Why not use this code as a basis?
- If the implementation has a CLI and a Web API (which is a good idea), I'm not sure if Dancer is the best solution for the Web API. In any case, the proper business logic should live in fat models, and be used by CLI and API "frontends".
To sum up, I think we should let ACT die, and either write something completly new from scratch, or maybe use some other conference toolkit (there have to be a lot!), even if they might not be written in Perl.
Of course it would also be a nice project to design a new ACT-like system, but IMO this HAS to be done by a team.
AFAIK there already were several hackathons trying to improve/rewrite ACT, which all failed. So this is not an easy task, and probably something better left to a "startup" - maybe a Perl community "startup", or a commercial one (but I think the marked for conference tools is quite flooded).
As a conference organiser and attendee, I'd also love a better Perl based tool, and I'd love to hack on one. But I think it's a too big job for one person...
The history records of performance issues and outages are certainly not great. And indeed it can cause much frustration for organisers and attendees when this is happening 'au moment suprême'.
The Dutch organising committee of TPC in Amsterdam had been challenged to move away from Act, it had a long period off outage and many more issues. But I had enough confidence in the people maintaining the service. Personally, I think, we do not give Sébastien, BooK and Laurent enough credit for the hard work the do behind the scenes.
Last year, due to a large migration project, some things did break, but were also fixed; Act is up and running ever since and but are quickly fixed when reported.
And with Act-out-of-the-Box, it will be more easy to make patches available, since current organizing team can work on things on their local machines.
Concerning the 'fact-finding-phase', I rather would leave that as something from the past. I am not sure that it will help us much in trying to figure out what went wrong there and then. I rather focus on a scalable solution that no longer needs the ancient system we need for legacy-act.
Once Act-Voyager is in a mature state, we can move on. We will have a better platform and technically only need a robust database. The old server can be shut down. - And for those that do have concerns about the websites from old days, yes, we will freeze them and serve them as static files.
This project would be very useful. ACT is a good tool but hard to get to grips with, so simplifying the process of getting started with it would be terrific.
Act-out-of-the-Box with the additional documentation on how to get started would be terrific move forward. Given Theo's very recent experiences with ACT, it seems a good time to have him work on this.
I do not think one should feel sorry to voice his concerns. Act-Voyager is a way forward to an entire new Act, but I would refrain using the term 'rewrite' for various reasons:
- The Database will mostly remain as they are, so the old legacy Act backend can continue working as it is
- The frontend will not be a 'rewrite' of what we have, it will be an entire new approach that relies on a REST-api ... whatever that front end will be, it most likely will not mimic the old websites, we are on warp-speed forward to modern web applications and more.
Writing a complete new product might make sense, but with this approach, we can live with both legacy and new next to each other.
Whatever one would call the objects and what other names live under the sun, my approach will deal with separation of business-object, presentation layer and storage.
My approach does mention te documentation of the REST-api, using the Open API Specification. This specifies how frontend and backend must communicate. It defines what the backend must adhere too, not how it has been implemented. How and where data has been stored is not of concern of a frontend client or CLI. But that also means that I do not have to define the exact objects being used, FAT Models, DAO's DBIC result sets. And, it also means I can use the current DataBase (which is not horribel) and if I must, I can swap the entire storage.... without breaking any APIO clients.
As far of failing hackathons, I only attended one of them. One can blame it on me, as everyone might have thought that I would arrange all. I think many will look back to that weekend in Lyon and will be able to pick things out of it that have been beneficial and learned a lot during these interesting discussions. I have not forgotten those 'take-home' experiences.
One thing I do have a lot of concern about is to make Act-Voyager accessible for everyone to play with and add valuable contributions to it:
- adding comments and patches to the OAS / Swagger file
- adding user stories related to the frontend for guests, users, organisers and admins
- help implementing and later
- add new features
Rather letting Act die, I would say it in a bit less radical way... put Act to REST!
the current DataBase (which is not horribel)
You're right. I guess the DB itself is ok (though I don't remember the schema in any detail), but the DB Abstraction Layer (which predates DBIx::Class and probably even Class::DBI) is horrible (at least I found it very hard to work with...)
Rather letting Act die, put Act to REST!
:-) But I still think that starting with a clean slate would do the project good. We do not need the old instances, we just need a dump if the current HTML (for future reference). Some of the old data might be handy (like attendee stats etc), but I don't think the valuable old data justifies keeping any compability with the old code.
Re: Swagger / Open API Spec
Have you actually used it in some real projects? I did, and while it has some merit in the early stages of a project, I've never seen a project actually use Swagger for the whole lifecycle, or for automatically creating clients etc. But maybe other people are more focused on Swagger than me, and do manage to achive what's promised on the tin :-)
And regarding REST / API etc: Of course ACT should be just a backend API wit various clients (commandline, JS Frontend, or even a Perl/TT-based "frontend"). But if we want to actually work on this idea, a "simple" rewrite / rebrush of the old code won't suffice.
I talked with BooK (one of the original authors) at TPCiA, and he had a few very good ideas for new / other features etc. You should definitly contact him, and/or get him to feedback here.
I do not think one should feel sorry to voice his concerns.
Act-Voyager is a way forward to an entire new Act, but I would refrain using the term 'rewrite' for various reasons:
- The Database will mostly remain as they are, so the old legacy Act backend can continue working as it is
- The frontend will not be a 'rewrite' of what we have, it will be an entire new approach that relies on a REST-api ... whatever that front end will be, it most likely will not mimic the old websites, we are on warp-speed forward to modern web applications and more.
Writing a complete new product might make sense, but with this approach, we can live with both legacy and new next to each other.
Whatever one would call the objects and what other names live under the sun, my approach will deal with separation of business-object, presentation layer and storage.
My approach does mention te documentation of the REST-api, using the Open API Specification. This specifies how frontend and backend must communicate. It defines what the backend must adhere too, not how it has been implemented. How and where data has been stored is not of concern of a frontend client or CLI. But that also means that I do not have to define the exact objects being used, FAT Models, DAO's DBIC result sets. And, it also means I can use the current DataBase (which is not horribel) and if I must, I can swap the entire storage.... without breaking any APIO clients.
As far of failing hackathons, I only attended one of them. One can blame it on me, as everyone might have thought that I would arrange all. I think many will look back to that weekend in Lyon and will be able to pick things out of it that have been beneficial and learned a lot during these interesting discussions. I have not forgotten those 'take-home' experiences.
One thing I do have a lot of concern about is to make Act-Voyager accessible for everyone to play with and add valuable contributions to it:
- adding comments and patches to the OAS / Swagger file
- adding user stories related to the frontend for guests, users, organisers and admins
- help implementing and later
- add new features
Rather letting Act die, I would say it in a bit less radical way... put Act to REST!
I endorse this project.
We had to face several server migration for Act in the last 2 years.
Each of them was painful because of the age of the code and techno, because we are not mastering the code and architecture.
From all the attempts to propose evolutions in Act or an alternative, Act Voyager is the only one still alive. I trust Theo to provide the Perl community with a better platform to support the conferences.
the current DataBase (which is not horribel)
There is already a perfect database abstraction layer available for the current schema. It was the first thing I did working on this project before. DBIx::Class is not only a nice ORM, it provides us also with many other features.
Interesting enough about the DBIC version is that I also introduced a new Result Object called 'Conference'. The entire ORM would implode lacking this, there should had been a relation between this object and most of the other result sets. The database schema will show all over the place a conf_id - but we had no such table for conferences.
Rather letting Act die, put Act to REST!
New features might need new databases, but this project is not initially set up for building new features. This will certainly make it more easy to add them. And there are dozens I can think of. Let it first get it up and running, make it better, make it faster and more importantly, make it more robust... Then add those new shiny features organisers and attendees dream off.
Re: Swagger / Open API Spec
There are some tools out there that nicely integrate with Swagger for testing and designing, even for setting up a mocked server. But one thing that is so important, is that the collective Open API files document the API describe the interface.
Based on those Swagger files, any developer can write a new frontend and any front end can communicate with whatever backend one would be implement.
There is an initiative for a Dancer2 plugin that would build the entire Dancer app based on a Swagger file, but that is in a very early stage.
And as far as I know, it should be quite easy to create a CLI client based on a Swagger file. But again, although these options are available, automatic generation of clients, server, running test, mocking a server... these tools are nice, but the definition of the API itself is more important.
Act 2
BooK has indeed many interesting ideas about Act2. Some of them I have been tempted to implement but would broaden the scope beyond the the boundaries of the Perl Community - and would not justify a Perl Foundation Grant. Act2 would be an interesting commercial project and become a competitor of MeetUp.
Bradley Andersen replied to comment from domm | September 26, 2017 11:43 AM
agreed.
- elohmrow (sorry, lost my login details)
I think Act and it's underlying data are worth quite a lot to the Perl community, and therefore worth investing in.
This project badly needs some modernization, and while this has been tried many times before (without success), I believe that the right way to actually get some movement forward is by having an invested and motivated person (like Theo) spend the necessary time to make it happen. Several people in the Perl community have tried rewriting from scratch, and all of them have failed to complete their effort, and my impression is that this is partly because of the magnitude of the problem. Act has a lot of valuable legacy and useful features, and re-implementing all of it in one go is only feasible for well-funded organizations.
As far as I'm concerned, the greatest deficit in the Perl community is the lack of available attention to spend on improving core infrastructure in the community. When someone offers to improve the situation, I'd say "go for it!" – the cost of doing nothing is much larger than the amounts we're talking about in this grant.
I'd support this grant proposal in a heartbeat.
A complete and working Act-out-of-the-Box would be a useful thing as currently the biggest problem with making any significant change is having to push the changes to the test/live servers and that means quick turn around isn't possible. And Act-out-of-the-Box would also make contributing fixes easier, as that's something I've been wanting to do.
I'm entirely unconvinced by anything else here and largely agree with domm.
Of note is that OpenAPI is currently in flux with the v2 to v3 thing and them breaking away from JSON Schema (we have discussed this in #swagger on irc.perl.org) so you're going to have legacy stuff before you even get going.
I can imagine people would doubt my motivation and passion about this project and have questions why it has not been done in the previous grant assignment.
It has been a year and a half that most of my energy has been taken up by organising The Perl Conference in Amsterdam. If anything I desired to do, it was getting Act-Voyager developed whilst organising the conference. Having had the basics setup and adding more and more features as organising the event would move on and require more of Act-Voyager. And not only would that have been for the TPCiA but it would also had applied with the Perl Conference in iAlexandria. However, organising TPCiA sucked up more energy then I anticipated.
During the conference in Amsterdam I was already planing on picking up Act-out-of-the-Box. And even after being told that the initial grant was revoked, I still did not give up on that.
Having spend many hours on that first thing and researching others aspects, writing additional modules, I am still highly motivated to cary out this important task that will benefit the Perl Community.
I think Salve is right when he says: «the right way to actually get some movement forward»
I know that Act-out-of-the-Box will be a huge benefit on itself. Over dinner, I spoke about the essence and simplicity that it would bring, together with Lance (thanks for the dinner). Therefor I have not stopped working on that and hope to have to get it out in the next week or so. Being «entirely unconvinced by 'anything else'» and «largely agreeing with Domm» does not make sense:
- having a REST-api
- having a CLI
- having a DBIx::Class abstraction layer on a not horrible database is not something Domm is disagreeing on, he has a different approach and want to kill current Act and start over.
And I say, the REST api gives an interface layer between any storage and the web client or CLI, allowing to let legacy Act and Act-Voyager live along side. Something very important whilst developing and events still using, and heavily relying on, the current Act.
I have personally maintained ACT web sites for 2 YAPC's, 1 TPC, and 6 or 7 PPW's. I would have welcomed this project 5 or 6 years ago. Now, I wonder if there isn't better places to spend grant money.
Recent conferences are starting to switch to commercial event management projects that provide the features that event-goers want today, mobile apps being a big part of that.
ACT did a lot of things great in the early days and that's primarily because it had many great people putting a whole lot of effort into it. But then, people ran out of steam. I can't blame them. They really put a lot of love into ACT.
One significant problem ACT has always had, and this project will not solve, is the diverse way that all of the Perl events need to use it. It takes a lot of work to transform the square peg that is a Perl conference into the round hold that ACT needs it to be. Staying tied to the same data structures will ensure this never gets fixed. Having ACT out of the box isn't really that great of a benefit. One of the main reasons ACT has been such a treasure in the past is the nature of the centralized data. Everybody's personal data is portable from one event to the next. You get to build a history of events you've been to. Meanwhile, event organizers also get to build a history of data from one event to the next. YAPC::NA::2012 in Madison decided not to use the same ACT server as everyone else. Instead, they stood up their own server, ran the event, and then took the site down. People reacted very badly to having to create yet another act account for Madison. And not having the site any more has caused a number of different problems both in terms of lost history and lost data. Once you make act out of the box easy, you actually make ACT lose one of its biggest selling points.
I wish Theo the best. I'm sure if anybody can fix it, it's him. I know he's put a lot of work into this already, and his work is very admirable. But I'm worried that after Theo puts a lot of love into ACT, and TPF puts a lot of money into it, it's still not going to match up with event-goers expectations. And then when Theo runs out of steam, we'll still be back to using off the shelf products again.
-Dan