Difference between revisions of "User:Tomato"

From ArchWiki
Jump to: navigation, search
(Modifying fileshareset scripts)
m (Log parser: fix formatting)
 
(29 intermediate revisions by the same user not shown)
Line 1: Line 1:
just my personal scrapbook
+
''just my personal scrapbook''
  
=== Configure file sharing from KDE4 using samba ===
+
== lessfs ==
Sadly, this functionality is implemented using SUID perl script, and as Archlinux does not support SUID scripts nor it has perl-suid package we will have to get our hands dirty.
+
=== Sizes of different databases in lessfs ===
 +
With a 64k blocksize administration overhead would be approx 5~10% of the size of blockdata.tch
 +
=== Good all-round mount options ===
 +
<pre>-o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino\
 +
      ,default_permissions,allow_other,big_writes,max_read=65536,max_write=65536</pre>
  
{{Note|Works with simple file sharing only at the moment, advanced file sharing still asks for root password}}
+
=== Filesystem, mounting but not working ===
 +
Each and every partition with datafiles need 10% of free space, lessfs will freeze all writes to the partition for as long as the free space isn't aviable.
  
Work plan:
+
Version 1.0.4 has new config parameters:
# install packages
+
<pre>
# configure samba
+
MIN_SPACE_FREE=2
# modify {{Codeline|fileshareset}} scripts
+
MIN_SPACE_CLEAN=5
# create C wrapper for {{Codeline|fileshareset}} to work as a SUID binary
+
CLEAN_PROGRAM=/tmp/clean.sh
# configure KDE side of things
+
</pre>
# share a folder
+
<code>MIN_SPACE_FREE</code> defines minimum space left (in percent?) on the device, <code>MIN_SPACE_CLEAN</code> defines the amount of space left that triggers the <code>CLEAN_PROGRAM</code>
  
Packages we will need:
+
=== Bucket sizes ===
* {{Package Official|kdebase-dolphin}} this is the interface we will use to share folders
+
Usually when performance drops after reaching a certain database size this indicates that the bucketsize selected in <code>/etc/lessfs.cfg</code> is to low. The default 'lessfs.cfg' example included with the code only allows the database to grow up to: 2GB after which performance degrades.
* {{Package Official|samba}} for the server that will share files
+
To store 50GB the bucketsize should be at least : 27262976
* {{Package Official|kdenetwork-filesharing}} for System Settings File Sharing part
+
<pre>
* {{Package Official|kdebase-runtime}} basic KDE workspace (but if you're reading this, chances are you already have it installed)
+
2MB bucketsize equals 1000000 database entries.
* {{Package Official|kdelibs3}} for {{Codeline|fileshareset}} (not sure about it, but as of KDE4.4.0 it's still a dependency)
+
To store 50GB :
* {{Package Official|gcc}} to compile SUID wrapper
+
50GB = 50*1024*1024*1024 = 53687091200 bytes
 +
53687091200/4096 (4k blocksize) = 13107200 entries in the databases.
 +
13107200/1000000 (divide by 10^6 entries)
 +
13 *2 (10^6 entries requires 2MB) = 26
 +
26 * 1024*1024 = 27262976 -> This is the bucketsize you need for 50GB with a 4k blocksize.
 +
</pre>
 +
(from http://sourceforge.net/projects/lessfs/forums/forum/932600/topic/3304815)
  
==== Installing requisites ====
+
If you ever need to change the _BS settings, just unmount lessfs, change the settings, mount lessfs and run defrag.
Install all packages and their dependencies:
+
  
{{Codeline|pacman -S kdebase-dolphin samba kdenetwork-filesharing kdebase-runtime kdelibs3 gcc}}
+
=== Files to put on different spindles ===
 +
For max performance compile with –with-sha3 and put the FILEBLOCK and BLOCKUSAGE databases each on there own raid one disk set. The faster these disks are the better. BLOCKDATA can be on raid5/6.
 +
(from http://www.lessfs.com/wordpress/?p=295)
  
Log out and log in again (kdelibs3 are in non standard place and so the <code>PATH</code> must be updated)
+
=== Log parser ===
 +
<pre>
 +
grep 'lessfs\[8118\]' /var/log/messages.log | grep -Eo \
 +
  'size .* bytes \: lzo compressed bytes .* and .* duplicate blocks' |\
 +
  awk ' { SUM += $2 ; SUMCP += $8 ; DUP += $10 } \
 +
  END { print "total size " SUM/1024/1024/1024 "GiB, compressed size " \
 +
  SUMCP/1024/1024/1024 "GiB (ratio 1 : " SUM/SUMCP ") with " DUP\
 +
  " duplicate blocks (" DUP*64/1024/1024 "GiB deduplicated)" }' </pre>
  
==== Configuring samba ====
+
change <code>lessfs\[8118\]</code> to the PID of lessfs you want to monitor and <code>DUP*64</code> to the block size in kiB you use in it.
A basic config file works OK, though you'll need to create Samba users (with <code>smbpasswd -a &lt;user name&gt;</code> as root):
+
{{File|name=smb.conf|content=[global]
+
workgroup=HOME
+
server string = Samba Server
+
log file = /var/log/samba/&m.log
+
max log size = 50
+
load printers = No
+
dns proxy = No
+
  
[homes]
+
== Configuring Samba Domain/File server. ==
comment = Home Directories
+
read only = No
+
browsable = No
+
browseable = No
+
}}
+
  
Alternatively you can configure samba with <code>security = share</code> and add <code>guest account = &lt;your user name&gt;</code> to get an anonymous Samba server
+
=== What we want ===
 +
basically, everything that Samba has to offer:
 +
* save and share files (duh)
 +
* edit files/folders ACLs to the maximum extent possible for Samba<br />http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/AccessControls.html
 +
** mapping of permission inheritance
 +
* save file attributes (archive, system, hidden, read only) on samba server without screwing UNIX permissions
 +
* have the ability to manage users/computers in domain without the need to use UID 0 (root) account<br />http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/rights.html
 +
* automatic creation of user folders
 +
* Previous Version share feature
 +
* enforcement of log-on hours, password complexity, password age and requests for password change, and group policies <br />http://wiki.samba.org/index.php/Implementing_System_Policies_with_Samba<br />http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/PolicyMgmt.html
 +
* view and set file system quota
 +
* time server/syncronisation
 +
* printing with generic drivers to CUPS (optional)
  
Remember to add samba to <code>DAEMONS</code> in <code>/etc/rc.conf</code> if you want it to start at boot time.
+
guide:
 
+
https://help.ubuntu.com/community/SettingUpSambaPDC
==== Modifying fileshareset scripts ====
+
{{Command|name=|output=for directory in /opt/kde/bin/ /usr/lib/kde4/libexec; do
+
  cd $directory
+
  sed 's/\/init.d\//\/rc.d\//g' fileshareset > fileshareset.pl
+
  rm filesharelist
+
  ln -s fileshareset.pl filesharelist
+
done}}
+
 
+
==== Create C wrapper ====
+
Create such files in your home directory:
+
 
+
{{File|name=kde4.c|content=#include <stdio.h>
+
#include <stdlib.h>
+
#include <string.h>
+
#include <sys/types.h>
+
#include <unistd.h>
+
 
+
#define EXEC "/usr/lib/kde4/libexec/fileshareset.pl"
+
 
+
int
+
main(int argc, char** argv)
+
{
+
  char **args;
+
 
+
  setegid(0);
+
  seteuid(0);
+
  setgid(0);
+
  setuid(0);
+
 
+
  args = calloc(argc+1, sizeof(char**));
+
  if (args == 0)
+
    exit(1);
+
 
+
  for (int i=0; i < argc; i++)
+
  {
+
    args[i] = calloc(1,strlen(argv[i]));
+
    if(args[i] == 0)
+
      exit(1);
+
    strcpy(args[i], argv[i]);
+
  }
+
  return execv(EXEC, args);
+
}
+
}}
+
 
+
{{File|name=kde3.c|content=#include <stdio.h>
+
#include <stdlib.h>
+
#include <string.h>
+
#include <sys/types.h>
+
#include <unistd.h>
+
 
+
#define EXEC "/opt/kde/bin/fileshareset.pl"
+
 
+
int
+
main(int argc, char** argv)
+
{
+
  char **args;
+
 
+
  setegid(0);
+
  seteuid(0);
+
  setgid(0);
+
  setuid(0);
+
 
+
  args = calloc(argc+1, sizeof(char**));
+
  if (args == 0)
+
    exit(1);
+
 
+
  for (int i=0; i < argc; i++)
+
  {
+
    args[i] = calloc(1,strlen(argv[i]));
+
    if(args[i] == 0)
+
      exit(1);
+
    strcpy(args[i], argv[i]);
+
  }
+
  return execv(EXEC, args);
+
}
+
}}
+
 
+
compile them using <code>gcc kde4.c -std=c99 -o kde4</code> and <code>gcc kde3.c -std=c99 -o kde3</code>.
+
Copy them to system folders:
+
{{Command|name=copying wrappers|output=cp kde3 /opt/kde/bin/fileshareset
+
chown root:root /opt/kde/bin/fileshareset
+
chmod u+s /opt/kde/bin/fileshareset
+
cp kde4 /usr/lib/kde4/libexec/fileshareset
+
chown root:root /usr/lib/kde4/libexec/fileshareset
+
chmod u+s /usr/lib/kde4/libexec/fileshareset
+
}}
+
 
+
==== Configure KDE ====
+
Right click any folder in your home directory and select Properties. Go to Share tab and click 'Configure File Sharing...', you'll be asked for root password.
+
(alternatively run <code>kcmshell4 fileshare</code> as root)
+
 
+
In the dialog that displays you need to select checkbox near "Enable Local Network File Sharing" and second one besides 'Simple sharing' (this one should be already selected).
+
 
+
Click 'Allowed Users' and select the second option -- one where users only from specified group can share files. Create new group, call it <code>samba-share</code>, add yourself, and any other user you might want to be able to share files.
+
 
+
Click OK until you close all the dialog boxes.
+
 
+
==== Share a folder ====
+
When the file sharing is configured you will see a new radio button on the Share tab -- Shared and Not shared. Change position to Shared, click OK. On the shared folder should appear now a small globe to indicate that it is shared.
+

Latest revision as of 11:40, 6 March 2010

just my personal scrapbook

lessfs

Sizes of different databases in lessfs

With a 64k blocksize administration overhead would be approx 5~10% of the size of blockdata.tch

Good all-round mount options

-o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino\
      ,default_permissions,allow_other,big_writes,max_read=65536,max_write=65536

Filesystem, mounting but not working

Each and every partition with datafiles need 10% of free space, lessfs will freeze all writes to the partition for as long as the free space isn't aviable.

Version 1.0.4 has new config parameters:

MIN_SPACE_FREE=2
MIN_SPACE_CLEAN=5
CLEAN_PROGRAM=/tmp/clean.sh

MIN_SPACE_FREE defines minimum space left (in percent?) on the device, MIN_SPACE_CLEAN defines the amount of space left that triggers the CLEAN_PROGRAM

Bucket sizes

Usually when performance drops after reaching a certain database size this indicates that the bucketsize selected in /etc/lessfs.cfg is to low. The default 'lessfs.cfg' example included with the code only allows the database to grow up to: 2GB after which performance degrades. To store 50GB the bucketsize should be at least : 27262976

2MB bucketsize equals 1000000 database entries.
To store 50GB :
50GB = 50*1024*1024*1024 = 53687091200 bytes
53687091200/4096 (4k blocksize) = 13107200 entries in the databases.
13107200/1000000 (divide by 10^6 entries)
13 *2 (10^6 entries requires 2MB) = 26
26 * 1024*1024 = 27262976 -> This is the bucketsize you need for 50GB with a 4k blocksize.

(from http://sourceforge.net/projects/lessfs/forums/forum/932600/topic/3304815)

If you ever need to change the _BS settings, just unmount lessfs, change the settings, mount lessfs and run defrag.

Files to put on different spindles

For max performance compile with –with-sha3 and put the FILEBLOCK and BLOCKUSAGE databases each on there own raid one disk set. The faster these disks are the better. BLOCKDATA can be on raid5/6. (from http://www.lessfs.com/wordpress/?p=295)

Log parser

grep 'lessfs\[8118\]' /var/log/messages.log | grep -Eo \
  'size .* bytes \: lzo compressed bytes .* and .* duplicate blocks' |\
  awk ' { SUM += $2 ; SUMCP += $8 ; DUP += $10 } \
  END { print "total size " SUM/1024/1024/1024 "GiB, compressed size " \
  SUMCP/1024/1024/1024 "GiB (ratio 1 : " SUM/SUMCP ") with " DUP\
  " duplicate blocks (" DUP*64/1024/1024 "GiB deduplicated)" }' 

change lessfs\[8118\] to the PID of lessfs you want to monitor and DUP*64 to the block size in kiB you use in it.

Configuring Samba Domain/File server.

What we want

basically, everything that Samba has to offer:

guide: https://help.ubuntu.com/community/SettingUpSambaPDC