CHANGES

Wednesday May 13th, 2015 (Home sick still... :( )
-------------------------------------------------

    Get the server sending back uncompressed position, orientation, linear, angular velocity
    for the one cube. I can verify that at least the client inputs are being sent across to server
    and being received, and then I can interpolate the position of the client

    Just take the latest position/orientation for the player cube received and render it

    Hey it's actually working. Holy shit! :)

    Client should not render cubes until it is active.

    Server is completely fucked up when I run in on the linode box though. Why?

    I'm fairly certain there was uninitialized data going on for the input packet serialize.

    Also it seems like the timing for the non-headless client is off causing it to fall behind and lose inputs.

    Disabled the delta encoding on client inputs because it seems to be broken in worst case.

    Reduce server send rate to 10 packets per-second. Make this configurable in const.h

    Client does not seem to display cube when snapshots are sent 10 packets per-second.

    There was uninitialized data in the snapshot packet. What the hell was I smoking yesterday? Sick I guess.

    Verify client now gets cube at 10HZ strobe. Yes. Ready for next stage.

    
Monday, May 11th, 2015 (Home sick from work)
--------------------------------------------

    Removed linux MOTD. Annoying.

    Get the non-headless client working and connect it to the server.

    Seems that the server gets into modes where the client drops inputs.

    I'm not sure if this is bursts of packet loss or the client being legitimately behind.

    If I set the client to run 16 ticks ahead of the server (safety) this would let me know
    if it is dropped packets or if the client is just late (due to packet loss or otherwise).

    Client still seems to be dropping a lot of inputs even at 16.

    I suspect the dropped inputs are not the network but are an error with the adjustment of client tick.

    Or they could be something caused by the non-headless client.

    Try with headless client. Does the same thing happen?

    It doesn't seem to. Not exactly making 60fps on client?


Sunday Lunch, May 10th, 2015
----------------------------

    Add adjustment sequence to input packet.

    Client send current adjustment sequence in input packet.

    Server should track first input for new adjust sequence.

    This way the server can defer measurement of an adjustment until after the inputs
    come back with that adjustment.

    This avoids feedback getting into the measurement!

    It seems to work.

    Would be great if the client would print out the # of ticks between client tick and server tick.

    This is the amount of resimulation that would be required for client side prediction.

    It's around 15 - 20 ticks over LAN. This seems reasonable.

    Client bracketing consistently to 0 consistently is a bit weird.

    Tried adjusting server to tick at 60HZ. Reduces from 15-20 to 12 ticks client side prediction.

    Seems reasonable.


Sunday Morning, May 10th, 2015
------------------------------

    Implement adjustment measure of minimum ticks ahead.

    Try implementing basic adjustment decision, eg. bump sequence, add adjustment integer

    Add adjustment sequence and adjustment to snapshot packet.

    Client should cache these values and detect changes.

    For the moment, client should make an immediate adjustment -- I think it will be unstable, but lets try it.

    First attempt it crashed and burned because it adjusted repeatedly.

    I've limited to just one adjustment and made sure that inputs are filled (temporary hack) so the server
    doesn't think that inputs are getting dropped because of holes in the input sliding window on client.

    Now measure input delivery ahead post-adjustment to see if it is adjusting correctly.

    It seems to work perfectly, however -- the measurement is clouded initially for a while
    until the client puts through inputs that have the adjustment.


Late Saturday Evening, May 9th, 2015
------------------------------------

    Added concept of bracketing.

    For now I just want the client to go into bracketing mode post-synchronize

    I don't think a recursive binary search is required.

    Just run with the post synchronize for a while.

    Work out what is the latest that inputs are delivered

    Then make that adjustment.

    The rest should just be handled via the small adjustments (speed up / slow down)

    I don't think synchronize sequence is required anymore. There is only one synchronize state.

    Removed sync sequence, bracket sequence.

    Did a bunch of stuff and now bracketing triggers a single adjustment. It seems to be basically perfect!

    Add some amount of safety to the bracketing, eg. 8 frames (1/30th of a second @ 240HZ)

    Implemented. Verified that the client is now delivering inputs with safety, however
    in certain circumstances you can get oscillation, eg. 4 frames on/off because there
    is jitter in packet delivery to server, eg. 2 frames one frame etc.

    This indicates that you shouldn't try to be perfect with the adjustment, just get it
    in the ballpark, and then make small adjustments if it is clearly too far ahead or behind.

    Client is sometimes sending 0 inputs.

    Added code to handle this and ignore.

    Also made sure that first input is the oldest input tick in the input packet, not the newest.

    Even so, still getting a few old inputs coming in after synchronize. 

    Could it be old input packets still in flight?

    Yes appears to be so.

    To ensure that only post-bracketed inputs arrive I added "bracketed" to the input packet
    and ignore bracketed while bracketing, and non-bracketed inputs after bracketing has finished.

    Seems to fix the dropped inputs.

    Client seems to drop first frame post becoming active, presumably because of lazy init in some physics stuff? Fuck...

    Fixed by fake pumping the world after init on client (world is still active at that point)

    Renamed "bracket adjustment" to "bracket offset" so it is not confused with adjustments 
    (small speed up slow down adjustments made while the client is active post-bracketing)

    Added client adjust data on server.

    Server has a per-client reconnect flag inside adjust data.

    Server sends that reconnect flag to client inside snapshot packet.

    Each time client drops an input, bump a counter. If the client exceeds count, set reconnect to true.

    If the client sees reconnect, start a reconnect.

    Done this and made snapshots not get through if client is in a connecting state.

    Verify this stuff all works.

    Find a way to simulate this, eg. fake dropped inputs.

    Added a 'suppress send packets' flag and set it for a brief period on the client.

    Seems that bracketing is not working properly on the reconnect post set client reconnect. Why?

    Found it. Bracketed was not getting cleared on client-side connect.

    If the client has not dropped inputs for n seconds, reset # of dropped inputs back to 0.

    This way it is not overly harsh of infrequent dropped inputs. They don't accumulate.

    Added #if RUN_TESTS on client and hooked up a test for forgetting dropped inputs. Works.


Saturday Evening, May 9th, 2015
-------------------------------

    Implement connect with connect sequence that increases so we can distinguish a connection 
    request for a reconnect from repeated packets for the same session.

    Verify that client can immediately reconnect.

    Client seems to be suddenly dropping a lot of inputs. Why?

    Something is broken. Even adding sync safety doesn't bring back from 0 inputs ahead.

    Something to do with connect w. the reconnect change?

    Nope. Just seems incorrect. Backing out agressive change from previously to sync closer.

    Fixed a bunch of logs where server thought client had dropped inputs.

    Fix is to not try to get inputs from a client who hasn't delivered an input yet.

    Next, ignore any inputs earlier than first input delivered by client.


Saturday Lunch, May 9th, 2015
-----------------------------

    Implement code to send all inputs up to maximum to server in each packet

    Confirmed working.

    On the server keep track of most recent input received.

    Add this most recent input to the snapshot packet as an input ack.

    Update client code to only send inputs that have not been acked yet.

    Update input packet to delta encode inputs with one bit if they have not changed.

    On the server add inputs to sliding window

    On the server get input from the client input sliding window before simulating tick.

    If an input is not there from a client when the server needs to simulate that tick,
    print out a warning and bump up a counter.

    Make sure warning is only sent to a client once they are connected and not synchronizing.

    Make sure warning only shows after client has first delivered an input!

    Added code to measure exactly how early the client is delivering inputs.

    The results is:

    "client 0 is delivering input 10 frames early"

    I think it's best to conservatively start with a highly buffered sync, and then to have the client
    speed up slowly to close this down (after all 10 frames is just, at 240HZ... ~4ms)

    Pulled it back without safety to 3 frames early.

    Ideally I think the sync should be conservative, and then small adjustments should be made from this point.


Saturday Morning, May 9th, 2015
-------------------------------

    Extend client to adjust client tick at point that synchronization ends
    to server tick plus offset, and start sending inputs to server.

    If server receives client input late, print out a log

    Client is delivering lots of inputs late. Sleep( 1 / 60 ) is not good enough.

    Clean up headless vs. non-headless client.

    Add break signal to headless client.

    Extend headless client so that it has proper frame timing like the server.

    The client delevering late inputs has been solved.

    Extend server so that it is aware that the client needs to deliver n inputs
    ahead of current tick, because it is about to simulate those! Therefore,
    client must be a bit further ahead.

    Extend synchronization on server to determine last acked input,
    when calculating offset, do it relative to oldest input received
    (eg. last recieved input + 1) rather than the input tick in the
    packet.

    To implement this server needs to track last tick received from client.

    Verified that client is delivering inputs ahead of client and introduced
    sync safety on the server side.

    Moving forward all calculations about how the server should synchronize
    time should be made on the server. Client should just do what it is told.


Monday Morning, May 4th, 2015
-----------------------------

    Implement code that measures difference between client and server tick
    during sync phase. initially just print it out.

    Extend client and server to both track ticks.

    Server now counts a number of samples and then ends sync stage.

    Client starts off not initially synchronizing (because cannot synchronize
    until the server tells the client the tick...), and client synchronize is
    kicked off by the server saying, OK we are synchronizing now.

    Added sync sequence so the server can discard old input sync packets.

    Client needs to get sync sequence from server.

    Server should ignore sync input packets from client without matching sync sequence.

    Server needs to communicate sync offset back to client.

    This probably means that server stays in sync until the client reflects back
    the expected sync offset once the maximum # of sync samples have been gathered.

    Extend the server to print out: "client synchronize offset: %d"

    Extend client to print out offset when detects synchronized.


Sunday, May 3rd, 2015
---------------------

    Client needs access to current tick, either in its tick or through a pointer to world.

    Server needs access to current tick as well.

    For now I think it is good to copy it across. I don't want client/server to manipulate
    the world directly, except through functions that accept the client/server and world 
    to bridge.

    Added client_tick and server_tick to client structure.

    Client tick is the last simulated tick from the client.

    Server tick is the most recently recieved tick from the server.

    I can use this to offset the client ticks by a fixed amount relative to the server.

    Once synchronization is complete the offset can be locked in and small adjustments made relative,
    eg. scale time so that you speed up or slow down by x ticks over y seconds.

    Bottom line the adjustments need to be small and in reality if late packets.

    Also, the adjustment should not be exact, eg. you have RTT this and jitter that.

    The adjustments should be driven by a binary thing only, which is -- end result: did all
    inputs get delivered on time? We don't even really need to calculate the average of input
    delay, we just need to find the first point where inputs are delivered.

    Bonus points this can be moved closer to the point where it starts to break, but then
    we should back off and deliver inputs at that breaking point plus a safety amount.

    Some safety amount is required in case of delay or random internet BS.

    The goal should be to deliver all inputs with an amount of safety.

    Perhaps just parameterize this safety in there and then aim for it directly. Binary search.

    While doing all this the connection should be sending bandwidth both directions that matches
    the expected worst case bandwidth of the connection up/down for this client, so there are no
    bufferbloat issues where latency goes higher under load.


Wednesday, April 29th, 2015
---------------------------

    Sketch out client data structure with state machine.

    Sketch out server data structure with client slots.

    Get the client detecting time out.

    Implement write packet function.

    Get the client sending client connection request packets while in connection request state.

    Implement read packet function.

    Work out how to process packets -- seems that we need a callback or in-place processing,
    either that or we need an allocator for packets according to type, eg. 

    Actually, kept it real simple. Just call out to an extern C function.

    Setup server to process connection request packet.

    Verify server actually gets the packet. Yes it does.

    Server is suddenly dropping a lot of frames. Why?!

    It's the simulation running slow in debug.

    Setup server to process connection accepted packet.

    Respond immediately with connection accepted packet / connection denied according to context.

    Verified client is receiving connection accepted packet.

    Setup client to process connection accepted, connection denied packet.

    Verify client is fully connected.

    Implement server timeout of clients

    Client should transition to sending input packets to server once fully connected

    Server should transition client from "connecting" to "connected" once the first input packet is received

    Server should start sending "snapshot" packets to client once client is fully connected.

    Pare back client and server logs to minimum set

    Just throw a rand() in the guid for now on client connect. Is it really necessary? Probably not.


Sunday Lunch, April 26th, 2015
------------------------------

    Design a protocol for client connect.

    Include reconnect concept. If the client is in a slot then we don't want that slot blocked until timeout.

    1. connection request: client guid (uint64_t)

    2. connection accepted: client guid, server guid (uint64_t)

    Once we have connection response the client should be sending input packets to the server continually (60 times per-second)

    Once we have first input packet the server should be replying back with snapshot packets (30 times per-second)

    Implement a system for serializing packets in packets.h

    Define all packet types required.


Sunday Evening, April 19th, 2015
--------------------------------

    Bring across enough networking (network.h) for the server to listen on UDP ports.

    This basically means address, socket class (stripped down).

    Where to store the server side of the connection?

    I think maybe a server struct is acceptable.

    Does the server live inside the world? No. I want the server to be separate from world,
    because world is shared between client and server. I want a server specific side to the
    connection vs. something shared.

    Added CTRL-C handling to server. Expect server will only ever run on macosx/unix.

    What's the next thing I need to do?

    Client needs to have a socket as well. 

    Client struct, client_init, client_update, client_term.

    Client should create socket with port 0. We don't care.

    Server needs a function to read and process packets.

    server_read_packets

    Client needs a function to send packets.

    client_send_packet

    Client also needs to read packets:

    client_read_packets

    Setup server to respond to client when each packet is received.

    Verified client and server and sending and receiving packets.


Sunday Morning, April 19th, 2015
--------------------------------

Setup a nice linux instance on www.terminal.com

This setup is nice because it makes it easy to stop/start linux instances
so they run only while I'm working on them, also it makes it very easy to
dynamically resize instances (add/remove CPU and memory) on the fly.

Seems the best option for the development of the server.


Saturday Morning, April 18th, 2015
----------------------------------

    Goal: Get the server running on EC2 and playing over the internet.

    Setup an EC2 instance.

    How much does an basic compute instance cost?

    Very little. Spot instance is 1 cent an hour? Seems reasonable.

    Got an instance setup and running the server, in short:

        1. install g++ (can't seem to get clang working on linux with premake4)
        2. install git
        3. setup key to access github for dev
        4. sync to repo
        5. build and install ode from source

    Polish up the ubuntu instance.

    alias "pm" to "premake4" (~/.bash_aliases)

    Fix up incompatibilies between GCC and clang version of server (macosx) in premake file.

    Remove the passphrase on the key I use to access github. 

    Annoying to have to enter the password everytime I want to access the git repo.


Friday off Work (PTO for denist in afternoon), April 18th, 2015
---------------------------------------------------------------

    Get existing code to do the authority walk working again.

    It should walk physics objects but mark the authority via the entity manager so it is generic,
    eg. not specific to cubes, but easily extended to all objects, and authority as a concept could
    exist for non-physics objects as well.

    Authority transfer is working again, and it seems much more robust now too.


Thursday Evening, April 17th, 2015
----------------------------------

    Added entity_index to add object for physics.

    This is necessary because authority interaction needs to walk physics object
    and get back to entity index, so we need a mapping physics_index -> entity_index.

    This is best done inside the physics manager.

    To do this need a mapping from physics index back to entity index.

    Added code to pass the entity index into the physics add cube.

    Seems that the render is not gamma correct? Linear? Why?

    Forgot to enable SRGB framebuffer.

    Increased max iterations to 64 because without that there is sliding and other bullshit.

    Will just have to deal with the additional CPU cost.

    Added some code to taper down the amount of bobbing force upwards. It's still vibrant though and higher than before.


Sunday Evening, April 12th, 2015
--------------------------------

    Added authority int and authority time to entity manager. 

    Seems overkill to have a separate authority manager.

    Fixed up cube game code to modify these authority via get/set authority on entity manager.

    Added code to force player objects to be forced to have their own authority each update.

    Quickly hack up code that colors cubes according to authority, ignoring blending.

    Whenever authority is set to a non-zero value, clear the authority time to 0.

    Implement code to walk over all cubes, for cubes that are at rest but have non-zero
    authority, increment authority time by dt.

    Once authority time exceeds AuthorityThreshold return to 0 authority and clear
    authority time.

    Fixed a bug with at rest that was breaking this. Setting object state each frame
    was happening, and it was clearing at rest flag for everything. 

    This probably also explains perf problems.


Sunday Afternoon, April 12th, 2015
----------------------------------

    Bring back player cube linear bobbing. Retune it.

    Bring back player cube angular bobbing. Retune it.

    Bring back player cube rotation to face the current direction of movement (eg. anticipation/work)

    Bring back the player stay upright torque. Retune it.

    Basically everything is tuned now. Not 100% perfect but pretty close. 

    Even improved the velocity tilt while in push mode to look better.


Sunday Morning, April 12th, 2015
--------------------------------

    Big initial hitch when the game starts. 

    Unfortunately, this may mean that we're CPU bound when all cubes are moving.

    Find out!

    Confirmed. This is actually bad news. It means we're CPU bound on the physics.

    Maybe Rubikon will be better...

    Cube blow seems to be pulling cube *down* towards floor? Why?

    Cause caused by cube blow origin being set to same position as the player cube, instead
    of being the projection of the player cube position onto the z=0 plane.

    Bring back linear drag. Required to slow the cube in the air while hovering.

    Linear drag is not applying because I'm not pushing back the object state
    from the cube entities to the simulation before physics update.

    Implement pre-physics update for cubes.

    Fixed a bunch of errors where initial entity state was not correctly setup.

    Remove scale from physics state. It's not part of physics state, it's part of physics init for object only.

    Now tune the player cube drag.

    Fixed by reducing player iterations down to 16. Was 64. 

    Figure there are enough ticks between render steps that less iterations per-tick are acceptable.

    Disable everything except the cube blow lift.

    Retune the jump and hover forces and drag. It's a bit draggy right now. Needs less drag. More spring.

    Got it basically back into the right area. Could probably use more fine tuning later.

    Getting some penetration with the player cube and the ground.

    May need to cosmetically enhance this, eg. if player cube center is below minimum z, clamp it.

    Fixed by making small adjustment to CFM


Saturday April 11th, 2015
-------------------------

    Setup the client so it also creates a world and updates it with ticks.

    Added code to get input on client using direct key state accessors.

    Doesn't seem that player cube entity is getting picked up by the entity manager,
    or at least, it isn't valid on get entity return?

    Validate cubes get added properly at entity level on setup?

    Was missing the set entity, plus some bugs in the entity allocate.

    Works now.

    Verified that input is getting properly sampled.

    Add code to physics manager to grab state from physics objects and update cube entities.

    Implement code to grab the render state from cube entities.

    Renamed RenderCubes to RenderState, it's now more generic. We could render more than cubes.

    Bunch of work converting over.

    Added render_get_state which takes a world struct. Generic.

    Actually instance the render object inside client.

    Bring across camera smoothing etc.

    Hook up basically minimal rendering code.

    It should in theory be rendering now but it's not. Why not?

    Not sure what is going on. Shader problems?

    Nope. Problems with render not having display width/height set up correctly.

    Cubes are now rendering in a strange way, but at least I'm seeing something.

    Why are cubes rendering like this? Seems like the rotation might be off -- x,y,z,w wrong order?

    Was just missing initialization for cube orientation, lin/ang vel.

    Also seems like the floor is not added properly in the simulation?

    Fixed. Added plane for floor.

    Adjusted drag so it's less strong. Was too much @ 240HZ.

    Seeing random stuff that looks like the delta time for frames is way off. eg. fast/slow.

    Some tuning of physics parameters, drag etc.


Thursday April 9th, 2015
------------------------

    Hook up the simulation so it runs on the server.

    Setup server ticks so we get 240HZ ticks on server running through simulation

    Move the simulation and entities code into shared, "World", eg. world struct

    Everything works from world struct. Don't have globals. Don't make this mistake.

    Make all code for setting up the world and pumping it shared in world.h


Tuesday April 7th, 2015
-----------------------

    Cleaned up the C++ bullshit in shader loading code


Sunday April 5th, 2015
----------------------

    Bring across the rest of the cube render into render.cpp

    Most brought across. 

    Convert the last glm::ortho to vectorial to remove that POS library.


Saturday March 28th, 2015
-------------------------

    Sketch out initial render.h

    Plan out what to bring across

    Seems that at minimum I really only need the load shader, unload shader.

    Don't really need the manager.


Friday March 27th, 2015
-----------------------

    Hook up player velocity tilt.

    Hook up player push bobbing etc.

    Hook up player push/pull on other cubes entities in the world.

    Game logic for processing player inputs and controlling the sim is now ported across.


Thursday March 26th, 2015
-------------------------

    Brought across update authority into entity.cpp (commented out)

    Bring across the game code into game.h. This is the code that takes player input and applies
    forces and so on to cubes. Structure this such that it is easily called within the context
    of client side prediction rollback.

    Ensure that game code doesn't talk to ODE directly. goes through physics.h

    Stopped work on this as more physics sim work needs to be done first.

    Extended base entity to have a physics index.

    Extended cube entity create to actually create a physics object and set "physics_index" in the object.

    Renamed base entity "index" to "entity_index" to avoid confusion.

    Extended physics update to accept tick #

    Can now continue with player input application...

    Hook up player strafe forces.

    Hook up player bobbing force and torque.


Wednesday March 25th, 2015
--------------------------

    Set up physics.h and physics manager. This is the physics data that shims to the simulation.

    For now bring across ODE, conversion to Rubikon comes later.

    Don't custom build a version of ODE it's simply not required. Use the brew version.

    Split up cpp into physics_ode.cpp so we can later on have a physics_rubikon.cpp implementation.

    Convert physics interface to use vectorial instead of shitty old math library.

    Clean up the physics interface.


Tuesday March 24th, 2015
------------------------

    Need to bring in vectorial for vec3, quat etc.

    Rename edict to entity. Entity is the name I need here. Edict is a strange influence from past projects. 

    Added entity type. Added cube type and null type.

    Setup cubes.h. 

    Each cube should then have its own cube index, and there should be a concept of "max cubes"

    Is this overkill? I don't think so. I need to be able to create cubes dynamically
    and track them separately from non-cube entities. It seems that per-entity type
    there should be a manager capable of iterating across those entities af that type
    directly, as well as owning the pool of those entities that can be allocated.

    This is the cube entity type and all related cube functionality.

    Sketch out basic functions, data structures in here. Seems reasonable.

    Setup default entity type to world.

    Since world entity is at zero and nothing special is done to set it up/load it, this works well.

    Eventually world entity should have a model, and this is the "static scene" portion of each level.

    While it should just derive from the base entity type, it will have its own manager
    "cube manager" which can strip mine cubes and provide a think per-cube if necessary.

    Actually implemented cube allocate and free. Should be functional now.

    Actually implement entity allocate.

    Hack up code to create the cube entities

    Add sequence # per-entity index.

    Implement entity free.

    Bump sequence # on entity free.


Monday March 23rd, 2015
-----------------------

    Create a simple edict like system, SoA that maps object indices to entities.

    0 is world.

    1 is player 1, etc.

    Rest of cubes are MaxPlayers.

    Work out relationship between edicts and entities.

    Are entities managed elsewhere?

    Who frees them?

    Who serializes them? :D

    I think the edict system just doesn't care. It has a type per-edict,
    and it is game codes responsibility to handle that type and serialize
    each type appropriately.

    We don't need dynamic types, eg. types not known at compile time.

    So a simple type per-edict is appropriate.

    First entity type will be cube entity.

    I think the cube manager owns the cubes, and inserts them into edict index.

    This way there is an owner, and the physics manager can have the cubes
    for fast iteration across, but also they live in the edict system and
    can be indexed as edicts with objects of other types.


Saturday March 21st, 2015
-------------------------

    Initial commit.

    Move constants into const.h

    Bring across protocol.h from compression demo

    Split out snapshot stuff from protocol.h

    Add constants to const.h for tick rate (240), server frame rate (30), client frame rate (60)

    Need to get an actual high precision timer for the server. I'm sure I have old code around there for this

    Bring across stuff for timers, sleep and so on (platform.h)

    Sketch out server loop with time and sleeps etc. Working fine.

    Sketched out a good server loop that measures jitter caused by sleep and works with it.

    Added code to detect dropped frames.

    Added safety code so that at least 50% of frame time is required to sim frame otherwise another frame is dropped.

    Setup premake for the client and server so I can link to stuff

    Bring across enough GLFW to get the display window created and updated

    Get the client main loop working

    Base it around VSYNC

    Make sure the client frame time is sampled immediately after frame swap so there is minimal variance

    Unfortunately quite a wide time variance exists... stupid non-realtime OS :p