In this lab, you will implement a simple locking-based transaction system in SimpleDB using page-level locking. You will need to add lock and unlock calls at the appropriate places in your code, as well as code to track the locks held by each transaction and grant locks to transactions as they are needed.
The remainder of this document describes what is involved in adding transaction support and provides a basic outline of how you might add this support to your database.
As with the previous lab, we recommend that you start as early as possible. Locking and transactions can be quite tricky to debug!
Quick jump to exercises:
For the first download, the easiest way to do this is to untar the new code in the same directory as your top-level simpledb directory, as follows:
$ cp -r cs133-lab2 cs133-lab4
$ wget https://www.cs.hmc.edu/~beth/courses/cs133/lab/cs133-lab4-supplement.tar.gz
tar -xvzf cs133-lab4-supplement.tar.gz
Now all files from lab 2 and lab 4 will be in the cs133-lab4 directory.
To work in Eclipse, create a new java project named cs133-lab4 like you did for previous labs.
In the remainder of this section, we briefly overview these concepts and discuss how they relate to SimpleDB.
transactionCompletecommand. Note that these three points mean that you do not need to implement log-based recovery in this lab, since you will never need to undo any work (you never evict dirty pages) and you will never need to redo any work (you force updates on commit and will not crash during commit processing).
You will implement
In the LockManager class, you will need to create data structure(s) that keep track of which locks each transaction holds and that check to see if a lock should be granted to a transaction when it is requested.
You will implement shared and exclusive locks; recall that these work as follows:
A transaction will need to acquire a shared lock on any page
before it reads it, and an exclusive
lock on any page before it writes it. You will notice that
we are already passing around
Permissions objects in the
BufferPool; these objects indicate the type of lock that the caller
would like to have on the object being accessed (we have given you the
code for the
indicates you need a shared lock, while
Permissions.READ_WRITE requires an exclusive lock.
If a transaction requests a lock that it should not be granted, your code should block, waiting for that lock to become available (i.e., be released by another transaction running in a different thread). The code for waiting (a "sleep") is implemented for you in the Lock Manager class.
LockManageris a private inner class within
Take a look at
BufferPool.getPage() and note that before getting the page for the caller,
getPage() first attempts to acquire the lock on the requested page using
You won't need
to add more code to
acquireLock() until Exercise 5, but you may wish to look at it briefly now
as it uses methods you will implement in this exercise.
For now, you will implement the core functionality in the Lock Manager class for getting and releasing locks.
LockManagerconstructor: Your constructor should create whatever data structure(s) you will be using to represent your lock table. The design of this is entirely up to you. You may decide to use multiple data structures, create a helper class, etc. As a design guideline, you should ensure your data structure(s) allow you to answer these questions:
lock(): To get a lock, this method first checks if the given transaction can acquire a lock using
locked(), which you will implement next. Right now you should just add code to
lock()to update your lock table assuming it's okay (this is the "else" case in the code).
locked(): This method returns a boolean indicating whether a transaction is "locked out" from acquiring a lock on the given page with the given permissions. Logic for this method appears in the code in comments above the method. Be careful with
equalsand be sure to check Java documentation for whatever data structures you are using for your lock table to see when methods can return
holdsLock(): Simple method used by Buffer Pool to determine whether the given transaction has any type of lock on the given page.
releaseLock(): Release whatever lock the given transaction has on the given page, updating your lock table. Used for testing (used by
BufferPool.releasePage()) and to help at the end of transactions, as you'll see in Exercise 4.
You may need to implement the next exercise before your code passes the unit tests in LockingTest.
Note: if it seems like LockingTest is hanging forever before it even runs any of its tests, the problem is likely happening in the setup() for LockingTest! Check out what happens there. In particular, does your implementation for
locked() allow a transaction to get a lock it already has? And allow a transaction to get a lock on a page if no other transaction holds a lock on that page?
Debugging tip: When running tests with ant, note that the printing of standard output will be delayed until the tests complete. If this is making debugging hard, you could try temporarily commenting out parts of the unit tests.
Now that you've implemented the core functionality for acquiring and releasing locks, you will need to implement strict two-phase locking. Recall that this means that transactions should acquire the appropriate type of lock on any object before accessing that object and shouldn't release any locks until after the transaction commits (or aborts).
Depending on your implementation, it is possible that you may not have to acquire a lock anywhere besides what you've already implemented in Buffer Pool. It is up to you to verify this in the next exercise!
You will need to think about when to release locks as well. It is clear that you should release all locks associated with a transaction after it has committed or aborted to ensure strict 2PL. You will implement this later in Exercise 4. However, it is possible for there to be other scenarios in which releasing a lock before a transaction ends might be useful. For instance, you may release a shared lock on a page after scanning it to find empty slots (as described in Exercise 2 below).
BufferPool.getPage(), this should work correctly as long as your
HeapFile.deleteTuple(), as well as the implementation of the iterator returned by
HeapFile.iterator()should access pages using
BufferPool.getPage(). Double check that that these different uses of
BufferPool.getPage()pass the correct permissions object (e.g.,
markDirty()on any of the pages they access (you should have done this when you implemented this code in Lab 2, but we did not test for this case.)
HeapFile. When do you physically write the page to disk? Are there race conditions with other transactions (on other threads) that might need special attention at the HeapFile level, regardless of page-level locking? You may need to synchronize the part of your code where you write a blank page to disk, e.g.,
In a NO STEAL policy, updates from a transaction cannot be written to disk before it commits. This means we must be sure not to evict dirty pages from the buffer pool until commit time.
Note that, in general, evicting a clean page that is locked by a running transaction is OK when using NO STEAL, as long as your lock manager keeps information about evicted pages around, and as long as none of your operator implementations keep references to Page objects which have been evicted. You don't need to do this for the lab.
This functionality is not tested until you've completed Exercise 4.
TransactionIdobject is created at the beginning of each query. This object is passed to each of the operators involved in the query. When the query is complete, the
Calling transactionComplete either commits or aborts the
transaction, as specified by the parameter flag
commit. At any point
during its execution, an operator may throw a
TransactionAbortedException exception, which indicates an
internal error or deadlock has occurred. The test cases we have provided
you with create the appropriate
TransactionId objects, pass
them to your operators in the appropriate way, and invoke
transactionComplete when a query is finished. We have also
At this point, your code should pass the
TransactionTest unit test and the
AbortEvictionTest system test. You may find the
TransactionTest system test
illustrative, but it will likely fail until you complete the next exercise.
It is possible for transactions in SimpleDB to deadlock due to a cycle of transactions waiting for each other to release locks. You will need to detect and resolve this situation!
There are many possible ways to detect deadlock. For example, you may:
After you have detected that a deadlock exists, you must improve the situation. Suppose you have detected a deadlock while transaction t is waiting for a lock; you can decide to abort t to give other transactions a chance to make progress. In the LockManager class, this is most easily done by aborting the xact t when it tries to acquire a lock that will cause a cycle (if you are trying the waits-for-graph approach) or if too much time has elapsed (if using the timeout approach).
BufferPool.java. If you are using the Lock Manager class, you will be checking for deadlocks (and possibly throwing a DeadLockException) in acquireLock().
Most likely, you will want to check for a deadlock whenever a transaction attempts to acquire a lock and finds another transaction is holding the lock (note that this by itself is not a deadlock, but may be symptomatic of one). For the waits-for-graph approach, you could check if a cycle of transactions waiting has formed. You have many decisions for your deadlock resolution system, but it is not necessary to do something complicated. Please describe your choices in the lab writeup.
Aborting a transaction
To abort a transaction in acquireLock, you can simply throw a DeadlockException. This is caught by BufferPool.getPage and converted to a
TransactionAbortedException, which in turn will be caught
by the code executing the transaction.
TransactionTest.java), which should call
transactionComplete() to clean up after the transaction.
You are not expected to automatically restart a transaction which
fails due to a deadlock -- you can assume that higher level code in the unit tests
will take care of this.
We have provided some (not-so-unit) tests in
DeadlockTest. They are actually a
bit involved, so they may take more than a few seconds to run (depending
on your policy). If they seem to hang indefinitely, then you probably
have an unresolved deadlock. These tests construct simple deadlock
situations that your code should be able to escape. The tests will
TransactionAbortedExceptions corresponding to
the deadlocks it successfully resolved to the console.
Note that there are two timing parameters near the top of
DeadLockTest.java; these determine the frequency at which
the test checks if locks have been acquired and the waiting time before
an aborted transaction is restarted. You may observe different
performance characteristics by tweaking these parameters if you use a
timeout-based detection method.
In addition to
your code should now should pass the
TransactionTest system test
(which may also run for quite a long time, but timing out at 10 minutes).
Debugging tip: TransactionTest runs actual queries--see the run method in XactionTester. So if DeadlockTest passes but TransactionTest does not, the issue may lie with the query plan operators that are used by this test, namely, Insert.java and Delete.java. If your implementation of the operators suppresses TransactionAbortedExceptions because it catches all exceptions, that could be the issue. Feel free to consult Lab 2 solution code.
For all deadlines besides the final version, you only need to submit cs133-lab4.tar.gz tarball (such that, untarred, it creates a cs133-lab4/src/java/simpledb directory with your code). You can use the ant handin target to generate the tarball.
For the final version of the lab, the files you need to submit are:
Submit all your files for Exercise 1-2 under "Lab 4: Part 1" and the final version of the lab under "Lab 4: Final" on Sakai.
Your grade for the lab will be based on the final version after all exercises are complete.
75% of your grade will be based on whether or not your code passes the system test suite we will run over it. These tests will be a superset of the tests we have provided. Before handing in your code, you should make sure produces no errors (passes all of the tests) from both ant test and ant systemtest.
Important: before testing, we will replace your build.xml and the entire contents of the test directory with our version of these files. This means you cannot change the format of .dat files! You should also be careful changing our APIs. You should test that your code compiles the unmodified tests. In other words, we will untar your tarball, replace the files mentioned above, compile it, and then grade it. It will look roughly like this:
$ tar xvzf cs133-lab4.tar.gz $ cd ./cs133-lab4 [replace build.xml and test] $ ant test $ ant systemtest [additional tests]
If any of these commands fail, we'll be unhappy, and, therefore, so will your grade.
An additional 25% of your grade will be based on the quality of your writeup, our subjective evaluation of your code, and on-time submission for the intermediate deadlines.