Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • qhong/crdt.el
  • jimmyhmiller/crdt.el
  • jming422/crdt.el
  • artenator/crdt.el
  • willforan/crdt.el
5 results
Show changes
Commits on Source (64)
Expand all Hide whitespace changes
Inline Side-by-side
HACKING.org
View file @ d9ea57d4
......@@ -6,7 +6,7 @@ This packages implements the Logoot split algorithm
~André, Luc, et al. "Supporting adaptable granularity of changes for massive-scale collaborative editing." 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing. IEEE, 2013.~
The CRDT-ID blocks are implemented by text property ='crdt-id=.
A continous range of text with the same ='crdt-id'= property represent a CRDT-ID block.
A continous range of text with the same ='crdt-id= property represent a CRDT-ID block.
The ='crdt-id= is a a cons of =(ID-STRING . END-OF-BLOCK-P)=,
where =ID-STRING= represent the CRDT-ID of the leftmost character in the block.
If =END-OF-BLOCK-P= is =NIL=, the block is a non-rightmost segment splitted from a larger block,
......@@ -14,62 +14,68 @@ so insertion at the right of this block shouldn't be merged into the block by sh
=ID-STRING= is a unibyte string representing a CRDT-ID (for efficient comparison).
Every two bytes represent a big endian encoded integer.
For base IDs, last two bytes are always representing site ID.
For base IDs, last two bytes are always representing Site ID.
Stored strings are BASE-ID:OFFSETs. So the last two bytes represent offset,
and second last two bytes represent site ID.
and second last two bytes represent Site ID.
* Protocol
Text-based version
(it should be easy to migrate to a binary version. Using text for better debugging for now)
Text-based version
(it should be easy to migrate to a binary version. Using text for better debugging for now)
Every message takes the form =(type . body)=
Note: Starting from =v0.3.0=, we separate /User IDs/ and /Site IDs/.
Site IDs are /buffer local/ and temporarily assigned to users with writable access.
Every message takes the form =(type . body)=
- Text Editing
A peer must obtain a =site-id= before performing the following operations,
by remote calling =crdt-get-write-access=. See [[Remote Function]].
+ insert ::
body takes the form =(buffer-name crdt-id position-hint content)=
body takes the form =(buffer-name user-id crdt-id position-hint content)=
- =position-hint= is the buffer position where the operation happens at the site
which generates the operation. Then we can play the trick that start search
near this position at other sites to speedup CRDT ID search
- =content= is the string to be inserted
+ delete ::
body takes the form =(buffer-name position-hint . crdt-id-list)=
body takes the form =(buffer-name user-id position-hint . crdt-id-list)=
- =crdt-id-list= is generated from =CRDT--DUMP-IDS= from the deleted text
- Peer State
+ cursor ::
body takes the form
=(buffer-name site-id point-position-hint point-crdt-id mark-position-hint mark-crdt-id)=
=(buffer-name user-id point-position-hint point-crdt-id mark-position-hint mark-crdt-id)=
=*-crdt-id= can be either a CRDT ID, or
- =nil=, which means clear the point/mark
- =""=, which means =(point-max)=
- =nil=, which means clear the point/mark
- =""=, which means =(point-max)=
- Contact information
+ contact ::
body takes the form
=(site-id name address port)=
when name is =nil=, clear the contact for this =site-id=
body takes the form =(user-id slot value)=
- =slot= can be one of
#+BEGIN_SRC emacs-lisp
name host service focus
#+END_SRC
+ focus ::
body takes the form =(site-id buffer-name)=
+ leave ::
body takes the form =(user-id)=
This message is sometime sent from client to server to indicate disconnection,
if the underlying proxy doesn't indicate disconnection properly.
- Login
+ hello ::
This message is sent from client to server, when a client connect to the server.
body takes the form =(client-name protocol-version &optional response)=
+ leave ::
This message is sometime sent from client to server to indicate disconnection,
if the underlying proxy doesn't handle it properly.
body takes the form =()=
body takes the form =(protocol-version &optional response)=
+ challenge ::
body takes the form =(salt)=
+ login ::
It's always sent after server receives a hello message.
Assigns an ID to the client
body takes the form =(site-id session-name)=.
Assigns a User ID to the client
body takes the form =(user-id)=.
- Initial Synchronization
+ sync ::
......@@ -97,6 +103,9 @@ and second last two bytes represent site ID.
This message is sent from server to client to notice that some messages from the
client is not processed due to error =(error-symbol . error-datum)=.
Normally client should follow initial synchronization procedure to reinitialize the buffer.
- =buffer-name= can also be =nil=, which signifies that it's a session error.
The only reasonable thing to do is to disconnect in this scenario.
Currently, this happens when client/server protocol version doesn't match.
- Buffer Service
+ add ::
......@@ -115,7 +124,7 @@ and second last two bytes represent site ID.
+ overlay-add ::
body takes the form
#+BEGIN_SRC
(buffer-name site-id logical-clock species
(buffer-name user-id logical-clock species
front-advance rear-advance
start-position-hint start-crdt-id
end-position-hint end-crdt-id)
......@@ -124,40 +133,49 @@ and second last two bytes represent site ID.
+ overlay-move ::
body takes the form
#+BEGIN_SRC
(buffer-name site-id logical-clock
(buffer-name user-id logical-clock
start-position-hint start-crdt-id
end-position-hint end-crdt-id)
#+END_SRC
+ overlay-put ::
body takes the form =(buffer-name site-id logical-clock prop value)=
body takes the form =(buffer-name user-id logical-clock prop value)=
+ overlay-remove ::
body takes the form =(buffer-name site-id logical-clock)=
- Remote Command
+ command ::
body takes the form =(buffer-name user-id logical-clock)=
- <<Remote Function>>
+ fcap ::
body takes the form =(fcap-symbol nonce in-states out-states . interactive-form)=
This grants a "functional capability" to a peer.
Nonce is a random number to prevent forging capability.
- =in-states= is a list of state symbols that the function depends on.
=out-states= is a list of state symbols that the function modifies and should be synchronized
to the caller.
See [[Allowed state symbols]].
+ funcall ::
body takes the form
#+BEGIN_SRC
(buffer-name spawn-site-id
site-id logical-clock state-list
command-symbol . args)
(user-id logical-clock spawn-user-id
state-list nonce fcap-symbol . args)
#+END_SRC
- =spawn-site-id= represents the site where the interactive command is originally invoked
+ It can be different from =site-id= because a remote command can call a remote command!
This is especially useful when client makes a remote call,
but the call on the server request some interactive input,
and such interactive call are remote-called back into the client.
- =state-list= is an alist of bindings.
(except that we use 1 element list for the CDRs, to save a dot in the serialized string)
(CDRs can also be 2 element list of the form =(crdt-id pos-hint)=)
Allowed symbols are
#+BEGIN_SRC
point mark mark-active transient-mark-mode last-command-event
#+END_SRC
- =spawn-user-id= represents the site where the interactive command is originally invoked
+ It can be different from =user-id= because a remote function can call a remote function!
This is especially useful when client makes a remote call,
but the call on the server request some interactive input,
and such interactive call are remote-called back into the client.
- =state-list= is an alist of bindings.
(except that we use 1 element list for the CDRs, to save a dot in the serialized string)
(CDRs can also be 2 element list of the form =(crdt-id pos-hint)=)
<<Allowed state symbols>> are
#+BEGIN_SRC
window window-point buffer buffer-content point
mark mark-active transient-mark-mode last-command-event
#+END_SRC
+ return ::
body takes the form =(site-id logical-clock state-list success-p . return-values)=
body takes the form =(user-id logical-clock state-list success-p . return-values)=
- Buffer local variables
+ var :: body takes the form =(buffer-name variable-symbol . args)=
......@@ -310,20 +328,17 @@ Q: What if Emacs GCs?
+ delete :: body takes the form =(buffer-name position length)=
- Peer State
+ cursor :: body takes the form =(buffer-name site-id point-position mark-position)=
+ cursor :: body takes the form =(buffer-name user-id point-position mark-position)=
=*-position= can be either an integer, or
- =nil=, which means clear the point/mark
+ contact :: same as primary protocol.
+ focus :: same as primary protocol.
+ leave :: same as primary protocol.
- Login
Note that we don't include challenge/response authentication mecahnism.
+ hello :: same as primary protocol.
+ leave :: same as primary protocol.
+ login :: same as primary protocol.
- Initial Synchronization
......
README.org
View file @ d9ea57d4
* Introduction
~crdt.el~ is a real-time collaborative editing environment for Emacs using Conflict-free Replicated Data Types.
~crdt.el~ is a real-time collaborative editing environment for Emacs
using Conflict-free Replicated Data Types.
Highlights:
- [[https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type][CRDT]], darling child of collaborative editing researches...
- [[https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type][CRDT]],
darling child of collaborative editing researches...
- Share multiple buffer in one session
- See other users' cursor and region
- Synchronize Org mode folding status
......@@ -16,112 +18,166 @@ Highlights:
~crdt.el~ is now on GNU ELPA! Just =M-x package-install crdt=.
*Caution!!!* Please make sure that you and your peers are on the same
~crdt.el~ version! It turns out to be one of the most common causes
of ~crdt.el~ not working. Because currently the network protocol is
not stablized, behavior when using mismatched versions is
unexpectable.
- Strictly speaking, it should work when =crdt-protocol-version= are
defined (added after version =0.2.5=) and the same on all peers.
But why not save some hassle and keep everyone on the latest
version.
- To upgrade, just =M-x package-reinstall crdt=, then preferably
restart Emacs. To check your ~crdt.el~ version, =M-x crdt-version=.
** Start a shared session
A shared session is a place that can contains multiple buffers (or files),
and multiple users can join to collaboratively edit those buffers (or files).
Think about a meeting room with some people working together on some papers.
A shared session is a place that can contains multiple buffers (or
files), and multiple users can join to collaboratively edit those
buffers (or files). Think about a meeting room with some people
working together on some papers.
In some buffer, =M-x crdt-share-buffer=. Then enter session name.
This add the current buffer to the existing session with that name.
If no such exists, it creates a new session with the provided session name,
and initially contains the current buffer as a shared buffer.
If no such exists, it creates a new session with the provided session
name, and initially contains the current buffer as a shared buffer.
If a new session is to be created, you need to enter port (default to
6530), optional password and your display name (default to your
current =(user-full-name)=).
If a new session is to be created, you need to enter port (default to 6530),
optional password and your display name (default to your current =(user-full-name)=).
Experimental settings: "Secure Port" specifies TLS port, and "Command
Functions" specifies user permissions. It's ok to just use the default
values.
** Join a session
=M-x crdt-connect=, then enter address, port, and your display name.
** What if we don't have a public IP?
If the server has provided the permission (this is the default case),
connected user may also add their buffers to the session via
=M-x crdt-share-buffer=.
There're various workaround.
Connection URLs look like =ein://ipaddress:6530=
- You can use [[https://gitlab.com/gjedeer/tuntox][tuntox]] to proxy your connection over the [[https://tox.chat][Tox]] protocol.
=crdt.el= has experimental built-in integration for =tuntox=.
To enable it, you need to install =tuntox=,
set up the custom variable =crdt-tuntox-executable= accordingly (the path to your =tuntox= binary),
and set the custom variable =crdt-use-tuntox=.
Setting it to =t= make =crdt.el= always create =tuntox= proxy for new server sessions,
and setting it to ='confirm= make =crdt.el= ask you every time when creating new sessions.
After starting a session with =tuntox= proxy,
you can =M-x crdt-copy-url= to copy a URL recognizable by =M-x crdt-connect= and share it to your friends.
Be aware that according to my experience, =tuntox= takes significant time to establish a connection (sometimes up to half a minute),
however it gets much faster after the connection is established.
** Navigate through sessions
- You can use Teredo to get a public routable IPv6 address.
One free software implementation is Miredo. Get it from your
favorite package manager or from [[https://www.remlab.net/miredo/][their website]].
A typical usage is (run as root)
#+BEGIN_SRC
# /usr/local/sbin/miredo
# ifconfig teredo
#+END_SRC
The =ifconfig= command should print the information of your IPv6 address.
Now your traffic go through IPv6, and once you start a =crdt.el= session,
your friends should be able to join using the IPv6 address.
For more information, see the user guide on the Miredo website.
- You can use SSH port forwarding if you have a VPS with public IP.
Example usage:
#+BEGIN_SRC
$ ssh -R EXAMPLE.COM:6530:127.0.0.1:6530 EXAMPLE.COM
#+END_SRC
This make your =crdt.el= session on local port =6530= accessible from
=EXAMPLE.COM:6530=.
- Sessions :: =M-x crdt-list-sessions= lists all sessions.
Note that you need to set the following =/etc/ssh/sshd_config= option on
your VPS
#+BEGIN_SRC
GatewayPorts yes
#+END_SRC
** List active users
In a CRDT shared buffer (either server or client), =M-x crdt-list-users=.
- Buffers :: =M-x crdt-list-buffers= lists all buffers in current
session. In the displayed buffer list, press ~RET~ in the session
list to see buffers in the selected session.
In the displayed user list, press ~RET~ on an entry to goto that user's cursor position.
Press ~f~ to follow that user, and press ~f~ again or =M-x crdt-stop-follow= to stop following.
You can also use =M-x crdt-switch-to-buffer= to interactively switch
to another buffer in the current session.
** List all sessions, and buffer in current session
- Users :: In a CRDT shared buffer (either server or client), =M-x crdt-list-users=
to list active users. In the displayed user list, press ~RET~ on an
entry to goto that user's cursor position. Press ~f~ to follow that
user, and press ~f~ again or =M-x crdt-stop-follow= to stop following.
=M-x crdt-list-sessions= lists all sessions.
=M-x crdt-list-buffers= lists all buffers in current session. Or you can also
press ~RET~ in the session list to see buffers in the selected session.
You can also use =M-x crdt-goto-next-user= and =M-x crdt-goto-prev-user=
to cycle through users' cursor positions from any CRDT shared buffer
(don't need to be in the user list buffer).
** Stop sharing
=M-x crdt-stop-session= stops a session you've started and disconnect all other users from it.
This will ask for your confirmation, customize =crdt-confirm-stop-session= if you want to disable it.
=M-x crdt-stop-session= stops a session you've started and disconnect
all other users from it. This will ask for your confirmation,
customize =crdt-confirm-stop-session= if you want to disable it.
You can also press ~k~ in the session list (show it by =M-x crdt-list-sessions=).
You can also press ~k~ or ~d~ in the session list (show it by =M-x crdt-list-sessions=).
=M-x crdt-stop-share-buffer= removes current buffer from its CRDT session
(this operation is only allowed at server side). Or press ~k~ in the buffer list.
=M-x crdt-stop-share-buffer= removes current buffer from its CRDT
session (this operation is only allowed at server side). You can also
press ~k~ or ~d~ in the buffer list.
** Disconnect from a session
=M-x crdt-disconnect=, then choose a session to disconnect from.
You can also press ~k~ in the session list (show it by =M-x crdt-list-sessions=).
You can also press ~k~ or ~d~ in the session list (show it by =M-x crdt-list-sessions=).
The server Emacs has the privilege to disconnect a user from a
session. To do so, press ~k~ or ~d~ on an entry in the user list
(show it by =M-x crdt-list-users=).
** Fancy stuff
- Visualizing author of parts of the document ::
Turn on =crdt-visualize-author-mode= to color text based on which
user authored it.
- Synchronizing Org folding status :: Turn on
=crdt-org-sync-overlay-mode=. All peers that have this enabled have
their folding status synchronized. Peers without enabling this minor
mode are unaffected.
- Comint integration ::
Just go ahead and share you comint REPL buffer! Tested: ~shell~ and
~cmuscheme~. By default, when sharing a comint buffer, ~crdt.el~
temporarily reset input history (as in =M-n= =M-p=) so others don't
spy into your =.bash_history= and alike. You can customize this
behavior using variable =crdt-comint-share-input-history=.
** What if we don't have a public IP?
There're various workaround.
- You can use [[https://gitlab.com/gjedeer/tuntox][tuntox]] to proxy
your connection over the [[https://tox.chat][Tox]] protocol.
=crdt.el= has experimental built-in integration for =tuntox=. To
enable it, you need to install =tuntox=, set up the custom variable
=crdt-tuntox-executable= accordingly (the path to your =tuntox=
binary), and set the custom variable =crdt-use-tuntox=. Setting it
to =t= make =crdt.el= always create =tuntox= proxy for new server
sessions, and setting it to ='confirm= make =crdt.el= ask you every
time when creating new sessions. After starting a session with
=tuntox= proxy, you can =M-x crdt-copy-url= to copy a URL
recognizable by =M-x crdt-connect= and share it to your friends. Be
aware that according to my experience, =tuntox= takes significant
time to establish a connection (sometimes up to half a minute),
however it gets much faster after the connection is established.
- You can use SSH port forwarding if you have a VPS with public IP.
Example usage:
#+BEGIN_SRC
$ ssh -R EXAMPLE.COM:6530:127.0.0.1:6530 EXAMPLE.COM
#+END_SRC
This make your =crdt.el= session on local port =6530= accessible
from =EXAMPLE.COM:6530=.
Note that you need to set the following =/etc/ssh/sshd_config=
option on your VPS
#+BEGIN_SRC
GatewayPorts yes
#+END_SRC
- For other tunneling solutions, see https://github.com/anderspitman/awesome-tunneling
* Service
=crdt.el= uses emacs' built in =(make-network-process)= to exchange elisp messages between clients, see =crdt--format-message=.
You can see these messages in within emacs with =(setq crdt--log-network-traffic t)= and outside emacs with =nc= or =telnet=.
#+begin_src bash
nc localhost 6530 <<< '(hello "0.3.0")'
#+end_src
The server Emacs has the privilege to disconnect a user from a session.
To do so, press ~k~ on an entry in the user list (show it by =M-x crdt-list-users=).
Demonstrating =(hello)=, =(get)=, and =(leave)=
#+begin_src bash
telnet localhost 6530
...
(hello "0.3.0")
** Visualizing author of parts of the document
Turn on =crdt-visualize-author-mode=. Colored underlines are added to each part of the document,
based on which user authored it.
(login 2)(add "testfile.org") ....
** Synchronizing Org folding status
(get "testfile.org")
(sync "testfile.org" (#("FILE CONTENTS ....
Turn on =crdt-org-sync-overlay-mode=. All peers that have this enabled have their
folding status synchronized. Peers without enabling this minor mode are unaffected.
(leave 2)
#+end_src
** Comint integration
** Messages
Just go ahead and share you comint REPL buffer! Tested: ~shell~ and ~cmuscheme~.
By default, when sharing a comint buffer, ~crdt.el~ temporarily reset input history (as in =M-n= =M-p=)
so others don't spy into your =.bash_history= and alike.
You can customize this behavior using variable =crdt-comint-share-input-history=.
hello, leave, get, cursor, overlay-move, overlay-remove, overlay-put, return, var, ready
crdt.el
View file @ d9ea57d4
This diff is collapsed.