The Reverend Joseph Williams, a native of the West Indies, came to Liberty County from Macon, Georgia, in the year 1867. On April 12, 1868, he organized a Presbyterian church in the building of "Old Midway" church with 300 members and worshipped there for eighteen years. This congregation became part of Knox Presbytery and of the Presbyterian Church U.S.A., and was known as the Midway Presbyterian Church.
Following a period of controversy over the rights of occupancy of the Old Midway-Congregational or Presbyterian the Reverend Joseph Williams organized a group of forty-six persons formerly members of a church pastored by Dr. C.C. Jones. In 1880 the church moved to Riceboro where a new building was constructed with the assistance of the Presbyterian Church U.S.A.
In 1887 the Ebenezer Presbyterian Church was organized at Freedman Grove. This church was also under the pastorate of the Reverend Joseph Williams.
In 1892 the congregation occupying the old Midway church, moved into Midway Chapel, which formerly served as a school building and in 1895 Midway Temple Presbyterian Church was built and dedicated. A loan of $1,000 from the Board of Erections, Presbyterian Church, U.S.A.; $200 in cash and $1,200 in labor, made this building possible. A shrine on the grounds of the Ebenezer Presbyterian Church in Freedman Grove marks the burial place of the Reverend Joseph Williams.